David Gardiner - GitHub Actions

Migrating my blog to Astro - Quality

2025-06-10T08:00:00.000+09:30

This is part of a series of posts on how I migrated my blog to Astro

There were a few things I put in place, or updated to improve the quality of the code and content of my blog.

Lychee link checking

I've blogged previously about using Lychee to find and fix broken links on your website. I figured this was a good time to revisit that process and see if it could be updated.

It turns out that Lychee recently changed their output format, so some of my GitHub Actions were no longer working. I've now fixed the Wayback machine query GitHub Action so it is compatible with Lychee 0.18 and above.

Here's an extract from my main GitHub workflow that runs Lychee:

  linkChecker:
    runs-on: ubuntu-latest
    needs: build
    name: Link Checker
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      # Cache lychee results (e.g. to avoid hitting rate limits)
      - name: Restore lychee cache
        uses: actions/cache@v4
        with:
          path: .lycheecache
          key: cache-lychee-${{ github.sha }}
          restore-keys: cache-lychee-

      - name: Download dist artifact
        uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist

      - name: Link Checker
        id: lychee
        uses: lycheeverse/lychee-action@v2.4.1 # 2.4.0 has a bug. Don't upgrade until fixed
        with:
          fail: false
          output: ${{ github.workspace }}/output.json
          format: json
          token: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
          args: --root-dir "$(pwd)/dist/" 'dist/**/*.html' --cache --max-cache-age 7d --max-retries 0 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:137.0) Gecko/20100101 Firefox/137.0" --fallback-extensions html -v --cache-exclude-status '429' --insecure --accept '200..=204'

      - name: Report results
        run: |
          ./Generate-LycheeReport.ps1 -InputJsonFile output.json -OutputMarkdownFile output.md

          ls -al

          cat output.md

          cat output.md >> $env:GITHUB_STEP_SUMMARY
        shell: pwsh

      - name: Publish results
        uses: actions/upload-artifact@v4
        with:
          name: link-checker-results
          path: output.*

I created a PowerShell script to generate a simple Markdown report that I could then output as a GitHub Actions summary.

The data file is also published as a build artifact.

I converted the links workflow so that it will now use that artifact as the source for creating a pull request with fixes for broken links where there is a match on archive.org.

Empty frontmatter

Sometimes I'd uncomment some of the frontmatter properties in a blog post, but forget to replace the placeholder text.

Whoops!

So I added an additional job to the main GitHub Actions workflow to validate frontmatter properties.

Lighthouse

Lighthouse is a popular tool for evaluating web page performance and accessibility.

I added a job to my GitHub Actions workflow to run a few key pages against Lighthouse.

The job publishes a GitHub step summary to the build, and links are provided if you want to drill into review the analysis of each page tested.

The results are generally very good. The main thing bring down the blog post page score is because it includes the Disqus commenting addin. One day I'll see if there's better alternative to that which plays nicer with the way it interacts with your web pages.

Markdownlint

I make regular use of David Anson's Markdownlint extension for VS Code, and so it made sense that I incorporate checking the entire set of Markdown files with a separate GitHub Actions workflow. I did have to do a bit of work to clean up some residual issues in older posts, but they're all fixed now and with this in place things should stay in good shape.

Automatic updating of Chocolatey packages with .NET

2024-03-31T17:00:00.000+10:30

I maintain quite a few Chocolatey packages. The source for these packages lives in https://github.com/flcdrg/au-packages/, and until recently I used the AU PowerShell module to detect and publish updated versions of the packages.

The first issue was that unfortunately, the original maintainer of the AU module archived the project on GitHub. The Chocolatey Community stepped in and is now maintaining a fork here.

The second issue I hit that was causing issues was a compatibility issue with newer versions of PowerShell 7. AU was originally written for Windows PowerShell 5, but I have made extensive use of features of PowerShell 6 and 7 in my update scripts. That didn't seem to cause issues until the GitHub Actions agents were updated from PowerShell 7.2 to 7.4 in January.

The specific problem would reveal itself like this:

Chocolatey had an error occur: System.ArgumentException: File specified is either not found or not a .nupkg file. 'D:\a\au-packages\au-packages\microsoft-teams.install\microsoft-teams.install.1.7.0.3653.nupkg '

For some reason, the AU module was able to generate a new version of a package, but when it called the Chocolatey CLI (choco.exe) and passed the path to the nupkg file, it appeared that there was a trailing space in the filename!

I spent hours trying to debug this to no avail. This was not made any easier by the fact that AU uses PowerShell Jobs to spin up separate processes for each package so they can be processed in parallel. I could not get debugging to work inside a Job when using the Visual Studio Code PowerShell debugger. Even the old-style debugging approach of Write-Host "I got here" didn't work very well as all output of the job is captured isn't easy to extract (let alone being able to inspect the original variables as proper objects rather than serialised strings)

Eventually, I decided I was wasting my time trying to solve this, and maybe if I rewrote the updating logic myself I could mitigate the issue.

There are essentially two parts to the AU module - the bits that support updating an individual package, and then there are the bits that run over all your packages. It's that second part that makes use of PowerShell Jobs and I suspected was the source of the problem.

I figured rewriting that part in C#/.NET would mean I had a much nicer debugging experience (should I need it). I wanted to leave the individual package updating alone - it would be a significant effort to migrate all the custom update.ps1 scripts to something else.

au-dotnet

And so au-dotnet was born.

It is a reasonably simple .NET 8 console application that iterates over all the packages in my au-packages repository, and then calls the PowerShell update.ps1 script in each to see if there is a new version to generate and publish.

Rather than just call out to the operating system to run each update.ps1 script, I decided to embed PowerShell in the application. This gives me a bit more control over how the scripts are run and the ability to capture any script output (and errors) from each run.

Hosting PowerShell

Figuring out how to host PowerShell in a .NET 8 application took a little bit of research. Many of the articles you find (and even some of the official documentation) are still aimed at Windows PowerShell.

The key was to reference these three NuGet packages (and use the same version of each package):

Microsoft.PowerShell.Commands.Diagnostics
Microsoft.PowerShell.SDK
System.Management.Automation

You can then create a PowerShell Class instance like this:

var iss = InitialSessionState.CreateDefault2();
iss.ExecutionPolicy = Microsoft.PowerShell.ExecutionPolicy.RemoteSigned;

var ps = PowerShell.Create(iss);

You can capture any output via the Streams property. eg. Here I am logging any errors from PowerShell as a GitHub Action error:

ps.Streams.Error.DataAdded += (_, args) =>
{
    core.WriteError(ps.Streams.Error[args.Index].ToString());
};

Running specific PowerShell cmdlets can be done via the AddCommand method. eg.

ps.AddCommand("Set-Location").AddParameter("Path", directory).Invoke();

Whereas running arbitrary PowerShell scripts is done via the AddScript method. eg.

ps.AddScript("$ErrorView = 'DetailedView'").Invoke();

If the script is in a separate .ps1 file, the only way I've found so far is to load that file into a string and pass it in. It would be nicer if you could point it at the file (so debugging/errors could include line numbers) but I have yet to find a way to do that.

var output = ps.AddScript(File.ReadAllText(Path.Combine(directory, "update.ps1")))
.Invoke();

One thing to remember is you must call the Invoke method to actually run the scripts or commands you've just added.

GitHub Action logging and summary

Because I know the application will be run in a GitHub Actions workflow, I made use of the excellent GitHub.Actions.Core NuGet package for formatting output, as well as generating a nice build summary that lists all packages that were updated in the current run.

Commit and publish

If a new package is created (eg. a .nupkg file now exists) then we assume this file can be submitted to the Chocolatey Community Repository. choco push is then called to upload the package. Remember this was where we hit that error with the trailing space? Pleasingly the .NET version doesn't exhibit this behaviour, so that problem is solved.

Assuming the package is submitted successfully then we call `git`` to stage any modified files from this package and add a tag indicating the package that was updated.

After all packages have been processed, we will commit all staged files and push the commit back to the repo, so that we get a version history of all the package changes.

Enhancements

I am currently using my fork of the chocolatey-au module, which has one minor enhancement. It adds a Files collection property to the AUPackage PowerShell class. This collection is populated with the paths of all the files that were downloaded (and had their checksums calculated).

I make use of this for some of my packages to pre-emptively upload the files to VirusTotal. This can help fast-track the packages being approved by Chocolatey as it means the Chocolatey virus scanning step is already completed. Because I use the VirusTotal CLI tool for this, it also means I can upload files up to 650MB (compared to Chocolatey's current 200MB limit due to using an older API).

I have submitted the Files property enhancement to chocolatey-au.

Summary

You can see this in action in the latest workflow runs at https://github.com/flcdrg/au-packages/actions.

Important changes for Azure DevOps Pipeline agents and GitHub Actions runners

2022-12-06T18:00:00.000+10:30

I don't think this has been publicised as widely as it should, especially for Azure Pipelines consumers. ubuntu-latest is now resolving to ubuntu-22.04 rather than ubuntu-20.04.

This has been mentioned in the GitHub blog and there is an 'announcement' issue in the runner-images repo, but I haven't seen anything official for Azure DevOps/Azure Pipelines. Both services use the same agent virtual machine images. As of this writing, the Microsoft-hosted agent page still suggests that ubuntu-20.04 is used.

This issue summarises the software differences between 20.04 and 22.04, and they can be significant. For example, I've already seen builds failing because they were assuming a .NET Core 3.1 SDK was preinstalled. The 22.04 image only includes .NET 6 and 7.

This is in addition to the already announced removal of ubuntu-18.04.

So make sure you declare all your tool requirements in your pipelines. Even better, run your jobs in a container (Azure Pipelines or GitHub Actions) with a custom container image that precisely specifies the tools and versions required to build and deploy your application.

Building and using Azure Pipelines Container Jobs

2022-07-20T09:00:00.000+09:30

The big advantage of using self-hosted build agents with a build pipeline is that you can benefit from installing specific tools and libraries, as well as caching packages between builds.

This can also be a big disadvantage - for example, if a build before yours decided to install Node version 10, but your build assumes you have Node 14, then you're in for a potentially nasty surprise when you add an NPM package that specifies it requires Node > 10.

An advantage of Microsoft-hosted build agents (for Azure Pipelines or GitHub Actions) is every build job gets a fresh build environment. While they do come with pre-installed software, you're free to modify and update to your requirements, without fear that you'll impact any subsequent build jobs.

The downside of these agents is that any prerequisites you need for your build have to be installed each time your build job runs. The Cache task might help with efficiently restoring packages, but it probably won't have much impact on installing your toolchain.

This can be where Container Jobs come into play.

Whilst this post focuses on Azure Pipelines, GitHub Actions also supports running jobs in a container too, so the same principles will apply.

Container jobs

Containers provide isolation from the host and allow you to pin specific versions of tools and dependencies. Host jobs require less initial setup and infrastructure to maintain.

You could use a public Docker image, like ubuntu:20.04, or you could create your custom image that includes additional prerequisites.

To make a job into a container job, you add a container, like this:

container: ubuntu:20.04

steps:
- script: printenv

With this in place, when the job starts, the image is automatically downloaded, a container using the image is started, and (by default) all the steps for the job are run in the context of the container.

Note that it's now up to you to ensure that the image you've selected for your container job has all the tools required by all the tasks in the job. For example, the ubuntu:20.04 image doesn't include PowerShell, so if I had tried to use a PowerShell@2 task, it would fail (as it couldn't find pwsh)

If the particular toolchain you require is closely tied to the source code you're building (eg. C# 10 source code will require at least .NET 6 SDK), then I think there's a strong argument for versioning the toolchain alongside your source code. But if your toolchain definition is in the same source code repository as your application code, how do you efficiently generate the toolchain image that can be used to build the application?

The approach I've adopted is to have a pipeline with two jobs. The first job is just responsible for maintaining the toolchain image (aka building the Docker image).

The second job uses this Docker image (as a container job) to provide the environment used to build the application.

Building the toolchain image can be time-consuming. I want my builds to finish as quickly as possible. It would be ideal if we could somehow completely skip (or minimise) the work for this if there were no changes to the toolchain since the last time we built it.

I've applied two techniques to help with this:

-cache-from

If your agent is self-hosted, then Docker builds can benefit from layer caching. A Microsoft-hosted agent is new for every build, so there's no layer cache from the previous build.

docker build has a -cache-from option. This allows you to reference another image that may have layers that be used as a cache source. The ideal cache source for an image would be the previous version of the same image.

For an image to be used as a cache source, it needs extra metadata included in the image when it is created. This is done by adding a build argument --build-arg BUILDKIT_INLINE_CACHE=1, and ensuring we use BuildKit by defining an environment variable DOCKER_BUILDKIT: 1.

Here's the build log showing how because the layers in the cache were a match, they were used rather than the related commands in the Dockerfile needing to be re-executed.

#4 importing cache manifest from ghcr.io/***/azure-pipelines-container-jobs:latest
#4 sha256:3445b7dd16dde89921d292ac2521908957fc490da1e463b78fcce347ed21c808
#4 DONE 0.9s

#5 [2/2] RUN apk add --no-cache --virtual .pipeline-deps readline linux-pam   && apk --no-cache add bash sudo shadow   && apk del .pipeline-deps   && apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib   && apk add --no-cache libgdiplus --repository https://dl-3.alpinelinux.org/alpine/edge/testing/   && apk add --no-cache     ca-certificates     less     ncurses-terminfo-base     tzdata     userspace-rcu     curl   && curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -Version 6.0.302 -InstallDir /usr/share/dotnet     && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet   && apk -X https://dl-cdn.alpinelinux.org/alpine/edge/main add --no-cache     lttng-ust   && curl -L https://github.com/PowerShell/PowerShell/releases/download/v7.2.5/powershell-7.2.5-linux-alpine-x64.tar.gz -o /tmp/powershell.tar.gz   && mkdir -p /opt/microsoft/powershell/7   && tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7   && chmod +x /opt/microsoft/powershell/7/pwsh   && ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh   && pwsh --version
#5 sha256:675dec3a2cb67748225cf1d9b8c87ef1218f51a4411387b5e6a272ce4955106e
#5 pulling sha256:43a07455240a0981bdafd48aacc61d292fa6920f16840ba9b0bba85a69222156
#5 pulling sha256:43a07455240a0981bdafd48aacc61d292fa6920f16840ba9b0bba85a69222156 4.6s done
#5 CACHED

#8 exporting to image
#8 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#8 exporting layers done
#8 writing image sha256:7b0a67e798b367854770516f923ab7f534bf027b7fb449765bf26a9b87001feb done
#8 naming to ghcr.io/***/azure-pipelines-container-jobs:latest done
#8 DONE 0.0s

Skip if nothing to do

A second thing we can do is figure out if we actually need to rebuild the image in the first place. If the files in the directory where the Dockerfile is located haven't changed, then we can just skip all the remaining steps in the job!

This is calculated by a PowerShell script, originally from this GitHub Gist.

Working example

I've created a sample project at https://github.com/flcdrg/azure-pipelines-container-jobs that demonstrates this approach in action.

In particular, note the following files:

Some rough times I saw for the 'Build Docker Image' job:

Phase	Time (seconds)
Initial build	99
Incremental change	32
No change	8

Additional considerations

The sample project works nicely, but if you did want to use this in a project that made use of pull request builds, you might want to adjust the logic slightly.

Docker images with the latest version should be preserved for use with main builds, or PR builds that don't modify the image.

A PR build that includes changes to the Docker image should use that modified image, but no other builds should use that image until it has been merged into main.

Is this right for me?

Measuring the costs and benefits of this approach is essential. If the time spent building the image, as well as the time to download the image for the container job exceeds the time to build without a container, then using a container job for performance alone doesn't make sense.

Even if you're adopting container jobs primarily for the isolation they bring to your builds, it's useful to understand any performance impacts on your build times.

Summary

In this post, we saw how container jobs can make a build pipeline more reliable, and potential improvements in time to build completion.

Fixing my blog (part 6) - Accessibility scanning

2022-04-24T09:00:00.000+09:30

Now we've got broken links sorted, we're in a better state to start accessibility testing using the Accessibility Insights Action. This is available as both a GitHub Action and an Azure DevOps extension. I'll be using the GitHub Action version.

name: Accessibility

on:
  workflow_dispatch:
    
jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - uses: ./.github/actions/jekyll-build-pages
        with:
          verbose: false

      - run: |
          sudo find -type f ! -regex ".*\.\(html\|svg\|gif\|css\|jpg\|png\)" -delete
        name: Remove non-HTML
        
        # https://github.com/microsoft/accessibility-insights-action
      - name: Scan for accessibility issues
        uses: microsoft/accessibility-insights-action@v2
        with:
            repo-token: ${{ secrets.GITHUB_TOKEN }}
            site-dir: ${{ github.workspace }}/_site
            scan-timeout: 6000000
            #max-urls: 1500
            localhost-port: 12345
            scan-url-relative-path: /

      - name: Upload report artifact
        uses: actions/upload-artifact@v2
        with:
            name: accessibility-reports
            path: ${{ github.workspace }}/_accessibility-reports/index.html

The action can either scan a local directory or a URL. In my case, I want it to scan all the files that make up my blog. My blog content is written in Markdown (.md) files and uses the Jekyll engine to render those pages into .html. It's the latter that should be scanned for accessibility compliance.

To generate the .html files, I make use of the Jekyll-Build-Pages action. This will generate a bunch of files under the _site directory.

My initial testing with the scanning tool revealed it was triggering on some pages I was including but had no control over (eg. Google Ads and the Disqus comments). I wanted to exclude those from the scanning, and one way to do that is to make the content conditional. eg.

{%- if jekyll.environment == "production" -%}
<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-999999999" crossorigin="anonymous"></script>
{%- endif -%}

To exclude this block, I then needed to ensure that jekyll.environment was not set to production. I achieved this by using a local copy of the action in which I set the value of the JEKYLL_ENV environment variable to development. To facilitate that I need to make a few other changes which you can see on this branch.

To help focus the scanning just on the files I care about, I added an extra step to the workflow to remove any files that weren't one of .html, .svg, .gif, .jpg or .png.

Depending on how many files you have in your website, scanning can take quite.

It's probably a good idea to not set max-urls the first time. This uses the default of 100, which will be enough to give you an idea of the kinds of problems you need to fix.

The reason for starting small is if you have a template or CSS that are used across every page, then every page will trigger errors, and your scan will take ages to finish and contains heaps of the same error(s).

Once you've resolved those common errors, then you can ramp up your max-urls to cover all the pages on your site (if it didn't already).

Running the workflow above also produces an 'Accessibility Checks' report. You can use this to get a quick overview of the results. To drill in to the details, you should download the build artifact view the index.html file in your browser.

Fixing the errors

The scan flagged 5 rule violations. Expanding each rule lists the URLs that exhibited the problem, plus a suggestion on how to resolve the issue. Sometime there are multiple suggestions.

color-contrast: Ensure the contrast between foreground and background colors meet WCAG 2 AA contrast ration thresholds

Example

<a href="/tag/Family.html">Family</a>

Fix the following:

Element has insufficient color contrast of 4.14 (foreground color: #2a7ae2, background color: #fdfdfd, font size: 12.0pt (16px), font weight: normal). Expected contrast ratio of 4.5:1Element has insufficient color contrast of 4.14 (foreground color: #2a7ae2, background color: #fdfdfd, font size: 12.0pt (16px), font weight: normal). Expected contrast ratio of 4.5:1
- Use foreground color: #2773d6 and the original background color: #fdfdfd to meet a contrast ratio of 4.58:1.
Element has insufficient color contrast of 3.77 (foreground color: #828282, background color: #fdfdfd, font size: 10.5pt (14px), font weight: normal). Expected contrast ratio of 4.5:1Element has insufficient color contrast of 3.77 (foreground color: #828282, background color: #fdfdfd, font size: 10.5pt (14px), font weight: normal). Expected contrast ratio of 4.5:1
- Use foreground color: #747474 and the original background color: #fdfdfd to meet a contrast ratio of 4.59:1.

These were common across all pages, so fixing these early is a quick win. I searched for the hex color (usually it was defined in one of the .scss files that generate the CSS) and replaced it with the suggested colour

link-name: Ensures links have discernible text

Fix the following:

Element is in tab order and does not have accessible text

Fix ONE of the following:

Element does not have text that is visible to screen readers
aria-label attribute does not exist or is empty
aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
Element has no title attribute

image-alt: Ensures `<img>` elements have alternate text or a role of none or presentation

Fix ONE of the following:

Element does not have an alt attribute
aria-label attribute does not exist or is empty
aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
Element has no title attribute
Element's default semantics were not overridden with role="none" or role="presentation"

html-has-lang: Ensure every HTML document has a lang attribute

Fix the following:

The <html> element does not have a lang attribute

This was a false positive as for some reason it was being flagged on images.

frame-title: Ensures `<iframe>` and `<frame>` elements have an accessible name

Fix ONE of the following:

Element has no title attribute
aria-label attribute does not exist or is empty
aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
Element's default semantics were not overridden with role="none" or role="presentation"

This was being flagged by some older YouTube embedded player HTML.

I've raise a bug to report the problem with links to images being flagged. It feels to me like the scanner is trying to scan image URLs (which it shouldn't). Whether that's a bug in the action, or in how I'm using it, I hope to find out soon.

Fixing my blog (part 5) - Putting it all together with a PR

2022-04-23T09:00:00.000+09:30

In part 4 of this series of posts we used the Replace multiple strings in files GitHub Action to update files with the replacement values. The files are modified on disk, but what to do with the changes? I could just automatically commit the changes back to version control, but I prefer to take a more cautious approach and give myself a chance to review the changes to confirm if they look reasonable. Creating a pull request is a great way of doing that.

A an action I've used successfully before to do this is Peter Evans' Create Pull Request GitHub Action. Using it is very easy:

- name: Create Pull Request
  uses: peter-evans/create-pull-request@v4.0.2
  with:
    token: ${{ secrets.PAT_REPO_FULL }}

I pass in a personal access token so that any workflows that should be run on creation of a pull request will be executed.

And with that, I have a full workflow for checking for broken links and repairing them. The only thing to watch out for is that issue I mentioned previously with the invalid JSON produced by the Lychee action. As a temporary measure, I inlined that action in my repo and applied a local code fix. Hopefully once the Lychee action itself is updated then I can go back to using their implementation.

name: Links

on:
  workflow_dispatch:
    
jobs:
  linkChecker:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
      - name: Link Checker
        id: lychee
        uses: ./.github/actions/lychee-action #lycheeverse/lychee-action@v1.4.1
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
        with:
          args: '--verbose ./_posts/**/*.md --exclude-mail --scheme http https'
          format: json
          output: ./lychee/links.json
          fail: false

      - uses: actions/upload-artifact@v3.0.0
        with:
          path: ./lychee/links.json

      - name: Wayback Machine Query
        uses: flcdrg/wayback-machine-query-action@v2
        id: wayback
        with:
          source-path: ./lychee/links.json
          timestamp-regex: '_posts\/(\d+)\/(?<year>\d+)-(?<month>\d+)-(?<day>\d+)-'

      - uses: actions/upload-artifact@v3.0.0
        with:
          path: ./wayback/replacements.json
          
      - name: Replacements
        uses: flcdrg/replace-multiple-action@v1
        with:
          find: ${{ steps.wayback.outputs.replacements }}
          prefix: '(^|\\s+|\\()'
          suffix: '($|\\s+|\\))'
          
      - name: Create Pull Request
        # if: ${{ github.ref == 'refs/heads/main' }}
        uses: peter-evans/create-pull-request@v4.0.2
        with:
          token: ${{ secrets.PAT_REPO_FULL }}

Now that we've addressed the broken links, we're in a better state to revisit the Accessibility Insights Action that started this whole adventure!

Fixing my blog (part 4) - Updating the files

2022-04-22T09:00:00.000+09:30

Last time we looked at using the Wayback Machine Query GitHub Action to automate querying the Wayback Machine for all our broken links. Now we need to apply the changes to our files.

Ideally there'd be a GitHub Action that could take a list of our changes and apply them to a list of files. I did search for something like that but all I could find were actions that only made one single change. Time to create another action.

Enter the Replace multiple strings in files GitHub Action.

For example, to find all instances of 'Multiple' and replace them with 'Many' for all the .md files in the current directory you can do:

- uses: flcdrg/replace-multiple-action@v1
  with:
    files: './*.md'
    find: '[{ "find": "Multiple", "replace": "Many" }]'

For my case I want something like this:

- uses: flcdrg/replace-multiple-action@v1
  with:
    files: './*.md'
    find: '[{ "find": "http://localhost", "replace": "https://localhost"}, { "find": "http://davidgardiner.net.au", "replace": "https://david.gardiner.net.au" }]'
    prefix: '(^|\\s+|\\()'
    suffix: '($|\\s+|\\))'

The prefix and suffix input properties need some explanation. Originally I was just using a plain string find and replace, but I discovered that there was a problem.

Consider that I could have multiple broken links to a site. eg. blog.spencen.com/2010/09/04/word-puzzle-to-sliverlight-phonendashpart-3.aspx and blog.spencen.com.

I replace all the instances of the first URL with https://web.archive.org/web/20100926212957/http://blog.spencen.com/2010/09/04/word-puzzle-to-sliverlight-phonendashpart-3.aspx.

The problem comes with the next find/replace in that it is now looking for blog.spencen.com and that value also exists in the new snapshot URL! We potentially end up in a recursive 'inception' mess. To avoid this, we can supply some partial regular expressions that get concatenation before and after the broken URL when we're searching. In my case those expressions mean that blog.spencen.com won't be matched in the middle of the snapshot URL.

I've made these properties so the action is more general than just my use case of updating my blog posts.

To actually use the action in concert with the previous actions, I'm using it thus:

- name: Replacements
  uses: flcdrg/replace-multiple-action@v1
  with:
    find: ${{ steps.wayback.outputs.replacements }}
    prefix: '(^|\\s+|\\()'
    suffix: '($|\\s+|\\))'

The key difference here is that I'm making use of the output property from the Wayback Machine Query action.

Our files have been updated. Next we finish off the workflow by creating a pull request with the changes.

Fixing my blog (part 3) - Querying the Wayback Machine

2022-04-21T09:00:00.000+09:30

Last time I'd discovered I had a huge link rot problem on my blog, but I didn't fancy copy and pasting all those broken URLs into the Internet Archive's Wayback Machine to search for the equivalent archive URL.

What I need is a way to query the Wayback Machine via some kind of API.. Like maybe this one?. The API is quite simple to use.

To look up the snapshot for Nigel's old blog site, I'd use the following request

https://archive.org/wayback/available?url=http://blog.spencen.com/

That returns the following JSON result

{
  "url": "http://blog.spencen.com/",
  "archived_snapshots": {
    "closest": {
      "status": "200",
      "available": true,
      "url": "https://web.archive.org/web/20201129195822/http://blog.spencen.com/",
      "timestamp": "20201129195822"
    }
  }
}

That's pretty good, but we can do better. An optional timestamp can be supplied in the query string (using the format YYYYMMDDhhmmss), so that instead of returning the most recent available capture, instead the snapshot closest to that timestamp will be returned. This is useful for URLs where the content might have changed over time. I realised that my blog posts include the date in the filename, so if I parsed the filename I could get a timestamp with a resolution of a specific day - that would be good enough to make the snapshot URLs more accurate.

So given the page that linked to Nigel's blog was named 2009-09-13-tech-ed-2009-thursday.md, then the query would become

https://archive.org/wayback/available?url=http://blog.spencen.com/&timestamp=20090913

And now we get this JSON result

{
  "url": "http://blog.spencen.com/",
  "archived_snapshots": {
    "closest": {
      "status": "200",
      "available": true,
      "url": "https://web.archive.org/web/20100315160244/http://blog.spencen.com:80/",
      "timestamp": "20100315160244"
    }
  },
  "timestamp": "20090913"
}

Browsing to that new snapshot URL shows a representation of Nigel's blog as it would have been when I wrote that blog post back in 2009.

So how can we automate this a bit more? By creating a new GitHub Action of course.

Enter the Wayback Machine Query GitHub Action!

If you look a the inputs for this action, it's no coincidence that the file it requires to provide the list of URLs to query the Wayback Machine with just happens to be compatible with the one generated by the Lychee Broken Link Checker GitHub Action we used in the previous post.

My new action returns the results in two output properties:

missing is an array of URLs that had no snapshots on the Wayback Machine. These URLs were never archived, so we'll have to deal with these differently.
replacements is an array of objects with the original URL and the Wayback Machine's snapshot URL.

The action also has an input property that allows us to provide a regular expression that can be used to parse the filename to obtain the timestamp.

Here's how I'm using it:

      - name: Wayback Machine Query
        uses: flcdrg/wayback-machine-query-action@v2
        id: wayback
        with:
          source-path: ./lychee/links.json
          timestamp-regex: '_posts\/(\d+)\/(?<year>\d+)-(?<month>\d+)-(?<day>\d+)-'

Obviously if your filenames or paths contain the date in a different format you'd need to adjust the timestamp-regex value (or if they don't contain the date then don't set that property at all).

We've now got a list of old and new URLs. In the next part we'll update the files.

Fixing my blog (part 2) - Broken links

2022-04-20T09:00:00.000+09:30

My first attempt to use the Accessibility Insights Action returned some actionable results, but it also crashed with an error. Not a great experience, but looking closer I realised it seemed to be checking an external link that was timing out.

Hmm. I've been blogging for a few years. I guess there's the chance that the odd link might be broken? Maybe I should do something about that first, and maybe I should automate it too.

A bit of searching turned up The Lychee Broken Link Checker GitHub Action. It turns out 'Lychee' is the name of the underlying link checker engine.

Let's create a GitHub workflow that uses this action and see what we get. I came up with something like this:

name: Links

on:
  workflow_dispatch:
    
jobs:
  linkChecker:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
      - name: Link Checker
        id: lychee
        uses: lycheeverse/lychee-action@v1.4.1
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
        with:
          args: '--verbose ./_posts/**/*.md --exclude-mail --scheme http https'
          format: json
          output: ./lychee/links.json
          fail: false

Results are reported to the workflow output, but also saved to a local file. The contents of this file look similar to this:

{
  "total": 5262,
  "successful": 4063,
  "failures": 1037,
  "unknown": 5,
  "timeouts": 125,
  "redirects": 0,
  "excludes": 32,
  "errors": 0,
  "cached": 381,
  "fail_map": {
    "./_posts/2009/2009-09-13-tech-ed-2009-thursday.md": [
      {
        "url": "http://blog.spencen.com/",
        "status": "Timeout"
      },
      {
        "url": "http://notgartner.wordpress.com/",
        "status": "Cached: Error (cached)"
      },
      {
        "url": "http://adamcogan.spaces.live.com/",
        "status": "Failed: Network error"
      }
    ],
    "./_posts/2010/2010-01-22-tour-down-under-2010.md": [
      {
        "url": "http://www.cannondale.com/bikes/innovation/caad7/",
        "status": "Failed: Network error"
      },
      {
        "url": "http://lh3.ggpht.com/",
        "status": "Cached: Error (cached)"
      },
      {
        "url": "http://www.tourdownunder.com.au/race/stage-4",
        "status": "Failed: Network error"
      }
    ],

(Note that there's a known issue that the output JSON file isn't actually valid JSON. Hopefully that will be fixed soon)

The fail_map contains a property for each file that has failing links, and for eacho of those an array of all the links that failed (and the particular error observed). Just by looking at the links, I know that some of those websites don't even exist anymore, some might still have the website but the content has changed, and some could be transient errors. I had no idea I had so much link rot!

Good memories Nigel, Mitch and Adam (the writers of those first three old sites)!

Ok, let's start fixing them.

But what do you replace each broken link with? Sure, some of the content might still exist at a slightly different URL, but for many the content has long gone. Except maybe it hasn't. I remembered the Internet Archive operates the Wayback Machine. So maybe I can take each broken URL, paste it into the Wayback Machine, and if there's a match, use the archive's URL instead.

Except I had hundreds of broken links. Maybe I could automate this as well?

Find out in part 3..

Fixing my blog (part 1) - Introduction

2022-04-19T13:00:00.000+09:30

I've been revisiting web accessibility. It's something I remember first learning about accessibility many years ago at a training workshop run by Vision Australia back when I worked at the University of South Australia. The web has progressed a little bit in the last 15 odd years, but the challenge of accessibility remains. More recently I had the opportunity to update my accessibility knowledge by attending a couple of presentations given by Larene Le Gassick (who also happens to be a fellow Microsoft MVP).

I wondered how accessible my blog was. Theoretically it should be pretty good, considering it is largely text with just a few images. There shouldn't be any complicated navigation system or confusing layout. Using tools to check accessibility, and in particular compliance with a particular level of the Web Content Accessibility Guidelines (WCAG) standard will not give you the complete picture. But it can identify some deficiencies and give you confidence that particular problems have been eliminated.

Ross Mullen wrote a great article showing how to use the pa11y GitHub Action as part of your continuous integration workflow to automatically scan files at build time. Pa11y is built on the axe-core library.

Further research brought me to Accessibility Insights - Android, browser and Windows desktop accessibility tools produced by Microsoft. From here I then found that Microsoft had also made a GitHub Action (currently in development) Accessibility Insights Action, which as I understand it, also leverages axe-core.

The next few blog posts will cover my adventures working towards being able to run that action against my blog. I thought it would be simple, but it turns out I had some other issues with my blog that needed to be addressed along the way. Stay tuned!

David Gardiner - GitHub Actions

Migrating my blog to Astro - Quality

Lychee link checking

Empty frontmatter

Lighthouse

Markdownlint

Automatic updating of Chocolatey packages with .NET

au-dotnet

Hosting PowerShell

GitHub Action logging and summary

Commit and publish

Enhancements

Summary

Important changes for Azure DevOps Pipeline agents and GitHub Actions runners

Building and using Azure Pipelines Container Jobs

Container jobs

-cache-from

Skip if nothing to do

Working example

Additional considerations

Is this right for me?

Summary

Fixing my blog (part 6) - Accessibility scanning

Fixing the errors

color-contrast: Ensure the contrast between foreground and background colors meet WCAG 2 AA contrast ration thresholds

link-name: Ensures links have discernible text

image-alt: Ensures <img> elements have alternate text or a role of none or presentation

html-has-lang: Ensure every HTML document has a lang attribute

frame-title: Ensures <iframe> and <frame> elements have an accessible name

Fixing my blog (part 5) - Putting it all together with a PR

Fixing my blog (part 4) - Updating the files

Fixing my blog (part 3) - Querying the Wayback Machine

Fixing my blog (part 2) - Broken links

Fixing my blog (part 1) - Introduction

image-alt: Ensures `<img>` elements have alternate text or a role of none or presentation

frame-title: Ensures `<iframe>` and `<frame>` elements have an accessible name