Verify.Cli - Snapshot testing without tests

28th July 2025

Verify.Cli is a new command-line tool for performing snapshot testing on individual files, without needing to create unit tests. Find out why I created this tool and examples of using it.

.NET

Unit Testing

In my last post, I looked at using the Verify library to implement snapshot testing in .NET unit tests.

Verify logo

I’ve been doing quite a lot of ‘DevOps’ work in recent years, and a large part of that has involved creating and optimising build and deployment (CI/CD) pipelines. Not infrequently I’ve wanted to add a sanity check to a pipeline to ensure a file has the correct content. “Oh if only I could call Verify from my pipeline”, I’d think.

If the file was in version control, I could sometimes fallback to relying on git diff to tell if the file had changed. But if it was a generated file that technique wouldn’t work. Other file comparison tools (like regular diff) might help, unless you had things like timestamps or other elements in the file that you wanted to ignore for the comparison. This is all bread and butter for Verify.

And so I created Verify.Cli!

Verify.Cli

Verify.Cli is a command-line tool that uses Verify internally. But while Verify was originally written to be used in unit tests, it is possible to host it in a console application and get similar functionality. It can be used to perform snapshot testing on an individual file.

You can install Verify.CLI as a .NET global tool, or there’s a Docker image so you can run it from a container.

Case study - feed.xml

I’ve recently rebuilt my blog using Astro. It generates a static website, but there is a challenge. Astro and the other libraries that are used to build my blog are updated frequently. It would be great to have confidence that those updates are not breaking the site. If it was a regular software application I could write some unit or integration tests. Given it generates static files, then the idea of snapshot testing some specific files would be helpful.

The feed.xml file is created when you run pnpm build in the dist folder. Here’s part of it:

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-AU" xmlns:media="http://search.yahoo.com/mrss/">
  <id>https://david.gardiner.net.au/feed.xml</id>
  <title type="html">David Gardiner</title>
  <updated>2025-07-26T10:39:59.564Z</updated>
  <subtitle>A blog of software development, .NET and other interesting things</subtitle>
  <generator uri="https://github.com/flcdrg/astrojs-atom" version="1.0.48">astrojs-atom</generator>
  <author>
    <name>David Gardiner</name>
  </author>
  <link href="https://david.gardiner.net.au/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://david.gardiner.net.au/" rel="alternate" type="text/html" hreflang="en-AU"/>
  <entry>
    <id>https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression</id>
    <updated>2025-07-14T08:00:00.000+09:30</updated>
    <title>Azure Pipelines template expressions</title>
    <link href="https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression" rel="alternate" type="text/html" title="Azure Pipelines template expressions"/>
    <category term="Azure Pipelines"/>
    <published>2025-07-14T08:00:00.000+09:30</published>
    <summary type="html">
      <![CDATA[Template expressions are a compile-time feature of Azure Pipelines. Learn how they differ to custom conditions
and see some common examples of their usage.]]>
    </summary>
    <content type="html" xml:base="https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression">
      <![CDATA[<p>In my <a href="/2025/06/azure-pipeline-conditionals">last post</a> I wrote about using custom conditions in Azure Pipelines to evaluate whether to skip a step, job or stage at runtime.</p>
<p>Sometimes we can do better though. With template expressions we can not just skip something, we can remove it entirely. We can also use it to optionally insert values in a pipeline (something you can't do with runtime custom conditions).</p>
<p>The important thing to remember is that template expression are a "compile time" feature. They can only operate on things that are available at compile time. <a href="https://learn.microsoft.com/azure/devops/pipelines/process/set-variables-scripts?view=azure-devops&amp;WT.mc_id=DOP-MVP-5001655">Variables set by scripts</a>, and <a href="https://learn.microsoft.com/azure/devops/pipelines/process/variables?view=azure-devops&amp;tabs=yaml%2Cbatch&amp;WT.mc_id=DOP-MVP-5001655#use-output-variables-from-tasks">task output variables</a> are two examples of things that are not available at compile time.</p>
<p>Compare these two Azure Pipeline runs. The first uses custom conditions to decided if the 'Publish Artifact' step is executed or not. Notice the 'Publish Artifact' step is listed, but the icon shown is a white arrow (rather than green tick)
<img src="https://david.gardiner.net.au/_astro/azure-pipelines-custom-conditions.Be6IaBQz_101Uki.webp" alt="Job showing a step 'Publish Artifact' that was conditionally not executed" /></p>
<p>If we use a template expression, then if it evaluates to false then the step is not even included in the job!</p>
<p><img src="https://david.gardiner.net.au/_astro/azure-pipelines-template-expressions.kRV13og-_2f5chf.webp" alt="Job without a 'Publish Artifact' step " /></p>
<p><a href="https://learn.microsoft.com/azure/devops/pipelines/process/template-expressions?view=azure-devops&amp;WT.mc_id=DOP-MVP-5001655">Template expressions</a> use the syntax <code>${{ }}</code></p>
<p>You can reference <code>parameters</code> and <code>variables</code> in template expressions. The latter are only variables that are defined in the YAML file and most of the <a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&amp;WT.mc_id=DOP-MVP-5001655">predefined variables</a>. (That page does list which variables can be used in template expressions, but you may need to scroll the page to the right to see that column!)</p>
<p>You can't reference variables that are created by scripts or anything else that is only available at runtime.</p>
<p>You can use <a href="https://learn.microsoft.com/azure/devops/pipelines/process/expressions?view=azure-devops&amp;WT.mc_id=DOP-MVP-5001655#functions">general functions</a> (the same ones we used previously with runtime Custom Conditions) in template expressions, as well as two special <a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/process/template-expressions?view=azure-devops&amp;WT.mc_id=DOP-MVP-5001655#template-expression-functions">Template expression functions</a>.</p>
<h2>Common patterns</h2>

We run the Verify CLI against this file like this (after creating a verified directory )

verify --file ./dist/feed.xml --verified-dir verified

Because there is no existing verified/feed.xml.verified.xml file, the Verify library launches Beyond Compare (my diff tool of choice) and I can use it to create the verified file.

However if I rebuild the site again and re-run verify, it shows up a difference. It turns out that the <updated>2025-07-26T10:39:59.564Z</updated> line is updated each time! Happily Verify’s scrubbers can handle timestamps like that.

If we were using Verify in a unit test, we’d configure this scrubber using VerifySettings.ScrubInlineDateTimes(). The equivalent for the command-line tool is --scrub-inline-datetime. If we run verify again using this:

verify --file dist/feed.xml --verified-dir verified --scrub-inline-datetime "yyyy-MM-ddTHH:mm:ss.fffZ"

We’ll see that the scrubber has replaced the timestamp with a placeholder:

  <updated>DateTime_1</updated>

With that in place we can repeat the validation with subsequent builds and be sure that the file should be equivalent. If the comparison ever fails in the future then we can be pretty confident it’s caused by a change in our logic or an update to one of the libraries.

Here’s how I’m using Verify.Cli in a GitHub Action workflow:

- name: Setup .NET
  uses: actions/setup-dotnet@v4
  with:
    dotnet-version: 9.x

- name: Install Verify.Cli
  run: dotnet tool install --global Verify.Cli

- name: Verify
  run: |
    verify --file dist/feed.xml --verified-dir verified --scrub-inline-datetime "yyyy-MM-ddTHH:mm:ss.fffZ"

Case study - blog post

The other thing I was keen to keep an eye on was individual blog posts. Here’s part of one:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <link rel="icon" href="/favicon.ico">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="generator" content="Astro v5.12.0">
    <link rel="canonical" href="https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression">
    <meta name="description" content="Template expressions are a compile-time feature of Azure Pipelines. Learn how they differ to custom conditions
and see some common examples of their usage."><meta name="og:description" content="Template expressions are a compile-time feature of Azure Pipelines. Learn how they differ to custom conditions
and see some common examples of their usage."><meta name="og:title" content="Azure Pipelines template expressions"><meta name="og:type" content="article"><meta name="og:url" content="https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression"><meta name="og:image" content="/_astro/azure-pipelines-logo.B45UakAg.png"><meta name="og:image:width" content="80"><meta name="og:image:height" content="80"><meta name="og:image:alt" content="Azure Pipelines logo"><script type="application/ld+json">{"@context":"https://schema.org","@type":"BlogPosting","mainEntityOfPage":{"@type":"WebPage","@id":"https://david.gardiner.net.au/2025/07/azure-pipeline-template-expression"},"headline":"Azure Pipelines template expressions","description":"Template expressions are a compile-time feature of Azure Pipelines. Learn how they differ to custom conditions\nand see some common examples of their usage.","image":"/_astro/azure-pipelines-logo.B45UakAg.png","author":{"@type":"Person","name":"David Gardiner","url":"https://david.gardiner.net.au/"},"datePublished":"2025-07-14T08:00:00.000+09:30"}</script>
    <title>Azure Pipelines template expressions</title>
    
  <link rel="stylesheet" href="/_astro/_slug_.CT3ttAlr.css"></head>
  <body>
    <header>
      <nav>
        <div class="nav-container" data-astro-cid-pux6a34n>

Most of that should be pretty stable. I tried a similar approach to what I used for feed.xml, and created a pull request then ran it in the GitHub Actions workflow… and it failed!

When I looked closer I realised that I’ve configured pull request build to use pnpm build --devOutput, whilst main branch builds just use pnpm build.

One difference with the former is that it adds an extra attribute to <img> elements - data-image-component="true". To be able to compare files we’ll need to remove all instances of that.

Verify.Cli has the --scrub-inline-remove parameter that we can use for that. eg.

--scrub-inline-remove ' data-image-component="true"'

When Astro processes images, it does appear to generate pretty stable URLs, but just in case it might be best to exclude those as well. Here’s a typical img tag:

<img src="/_astro/MVP_Logo_Horizontal_Preferred_Cyan300_RGB_300ppi.a-4xiaYx_2bYEEr.png"

These are not fixed strings - they are unique for each image, so using a regular expression would make sense. This time we’ll use --scrub-inline-pattern with an appropriate expression:

--scrub-inline-pattern '(?<prefix>")/_astro/[^"]+(?<suffix>")'

This parameter has an extra feature that I’m taking advantage of. If you include named groups in the regular expression ‘prefix’ and or ‘suffix’, then the scrubber will add those back to the start and end. I’m using that so that the double quotes are maintained.

The replacement looks like this:

<img src="STRING_5"

Here’s the full command:

verify --file dist/2025/07/azure-pipeline-template-expression.html --verified-dir verified --scrub-inline-pattern '(?<prefix>")/_astro/[^"]+(?<suffix>")' --scrub-inline-remove ' data-image-component="true"'

In this case, the GitHub Action workflow is using the Docker image to run Verify.Cli (another way of using it instead of installing it as a .NET global tool):

- name: Verify post
  run: docker run --rm -v $PWD:/tmp flcdrg/verify-cli --file /tmp/dist/2025/07/azure-pipeline-template-expression.html --verified-dir /tmp/verified --scrub-inline-pattern '(?<prefix>")/_astro/[^"]+(?<suffix>")' --scrub-inline-remove ' data-image-component="true"' --verbosity detailed

What if it fails?

If Verify.Cli fails locally on your machine, you can quickly identify the conflict via your visual diff tool of choice. But if it fails in a pipeline, then currently all you get is an error that the files are different. For now, I’ve come up with a hacky workaround:

- name: Diff files if error
  if: failure()
  run: |
    diff -u dist/feed.xml verified/feed.xml.verified.xml

This step in the workflow only runs if the Verify.Cli step failed. It assumes you have the diff tool installed on your build agent. This could be an aspect to improve in Verify.Cli in the future.

Conclusion

Take a look at https://github.com/flcdrg/astro-blog-engine for a repo making use of Verify.Cli (including the main workflow).

If you have a need to use the snapshot testing technique on individual files outside of unit tests, please give Verify.Cli a go. Let me know in the comments what you think, and feel free to submit any suggestions or bug reports over at the repo.

Helmet by Leonidas Oikonomou from Noun Project (CC BY 3.0)