WebSub notifications when my blog has new content

Using the WebSub protocol to notify interested parties that there's new content on my blog

Blogging

GitHub Actions

When I write a new post here on my blog, it updates the Atom (RSS) feed. But how does anyone know that the feed has changed?

If you’re using some kind of desktop RSS Feed reader, then it may just check all the RSS feeds when you open it, or maybe at a specified interval. Likewise if you use an online service like Feedly, then presumably they also are checking feeds regularly. I imagine they’re probably watching how often a feed changes and then adjusting the frequency of checking.

But wouldn’t it be nice if I could just tell Feedly (or anyone else for that matter) that I’ve published this new post, and then anyone who was interested could then retrieve the latest content immediately?

I wondered if Feedly might have some kind of endpoint that I could ping. I found this page in their documentation that mentioned something called “PuSH protocol”. I did find https://push.org/docs/ but that’s something to do with blockchain and I didn’t think was related.

I then resorted to asking Copilot and it suggested that “PuSH” was “short for PubSubHubbub, now standardised as WebSub”.

PubSubHubbub can still be found at https://github.com/pubsubhubbub, and the W3C specification for WebSub is at https://www.w3.org/TR/websub/. It’s amazing that this is not brand new - it’s been around for at least 10 years!

The WebSub specification describes a system with “publishers”, “subscribers” and “hubs”. Publishers tell the hub when their content has changed. The hub then notifies all subscribers of that content that there’s been a change.

The two most popular hubs seem to be Superfeedr and Google PubSubHubbub Hub. It turns out I already had an account with Superfeedr - I think I might have created it a few years ago when I was researching alternatives to Feedburner.

Superfeedr don’t charge for publishers, but they do for subscribers. I’m only a publisher so that’s fine.

Google’s site looks like it hasn’t changed in 10 years, and you never can tell with Google how long they’re going to keep running something. But it doesn’t hurt to cover my bases, so after originally just using Superfeedr I decided to also add Google’s hub too.

There were two things I needed to change in my blog to support WebSub:

Update Atom feed

The Atom feed for my blog needs to have link elements added to the feed section. You need one with rel="self" that points to your feed, and then you need a rel="hub" for each hub that you want to act as a publisher for:

<link href="https://david.gardiner.net.au/feed.xml" rel="self" type="application/atom+xml"/>
<link href="https://flcdrg.superfeedr.com" rel="hub"/>
<link href="https://pubsubhubbub.appspot.com/" rel="hub"/>

Ping the hub(s)

With that in place you then need to ping the hubs each time your feed changes. As I’m using GitHub Action workflows I added these jobs to do that:

  ping_superfeedr:
    runs-on: ubuntu-latest
    needs: publish_to_cloudflare
    name: Ping Superfeedr

    steps:
      - shell: pwsh
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: |
          $url = "https://david.gardiner.net.au/feed.xml"
          Write-Host "Pinging Superfeedr with URL: $url"

          $response = Invoke-WebRequest -Method Post -Uri "https://flcdrg.superfeedr.com" -Body @{ "hub.mode" = "publish"; "hub.url" = $url }
          Write-Host "Response: $($response.StatusCode)"

  ping_pubsubhubbub:
    runs-on: ubuntu-latest
    needs: publish_to_cloudflare
    name: Ping PubSubHubbub

    steps:
      - shell: pwsh
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: |
          $url = "https://david.gardiner.net.au/feed.xml"
          Write-Host "Pinging PubSubHubbub with URL: $url"

          $response = Invoke-WebRequest -Method Post -Uri "https://pubsubhubbub.appspot.com/" -Body @{ "hub.mode" = "publish"; "hub.url" = $url }
          Write-Host "Response: $($response.StatusCode)"

It’s interesting that Superfeedr creates a custom domain for your account to use https://flcdrg.superfeedr.com/, whereas Google just have their main domain that is used by everyone.

You can see both of these jobs in the context of my blog’s full workflow here

Summary

The end result is that now Feedly shows my post almost immediately after it is published. Any other subscribers will have the same experience.

Nice!

Now if only I could get Google to resume indexing my blog again. It suddenly stopped doing it back in May and everything I’ve tried so far since has not changed that. At least Bing and by extension DuckDuckGo still like me.