David Gardiner - Chocolatey

Automatic updating of Chocolatey packages with .NET

2024-03-31T17:00:00.000+10:30

I maintain quite a few Chocolatey packages. The source for these packages lives in https://github.com/flcdrg/au-packages/, and until recently I used the AU PowerShell module to detect and publish updated versions of the packages.

The first issue was that unfortunately, the original maintainer of the AU module archived the project on GitHub. The Chocolatey Community stepped in and is now maintaining a fork here.

The second issue I hit that was causing issues was a compatibility issue with newer versions of PowerShell 7. AU was originally written for Windows PowerShell 5, but I have made extensive use of features of PowerShell 6 and 7 in my update scripts. That didn't seem to cause issues until the GitHub Actions agents were updated from PowerShell 7.2 to 7.4 in January.

The specific problem would reveal itself like this:

Chocolatey had an error occur: System.ArgumentException: File specified is either not found or not a .nupkg file. 'D:\a\au-packages\au-packages\microsoft-teams.install\microsoft-teams.install.1.7.0.3653.nupkg '

For some reason, the AU module was able to generate a new version of a package, but when it called the Chocolatey CLI (choco.exe) and passed the path to the nupkg file, it appeared that there was a trailing space in the filename!

I spent hours trying to debug this to no avail. This was not made any easier by the fact that AU uses PowerShell Jobs to spin up separate processes for each package so they can be processed in parallel. I could not get debugging to work inside a Job when using the Visual Studio Code PowerShell debugger. Even the old-style debugging approach of Write-Host "I got here" didn't work very well as all output of the job is captured isn't easy to extract (let alone being able to inspect the original variables as proper objects rather than serialised strings)

Eventually, I decided I was wasting my time trying to solve this, and maybe if I rewrote the updating logic myself I could mitigate the issue.

There are essentially two parts to the AU module - the bits that support updating an individual package, and then there are the bits that run over all your packages. It's that second part that makes use of PowerShell Jobs and I suspected was the source of the problem.

I figured rewriting that part in C#/.NET would mean I had a much nicer debugging experience (should I need it). I wanted to leave the individual package updating alone - it would be a significant effort to migrate all the custom update.ps1 scripts to something else.

au-dotnet

And so au-dotnet was born.

It is a reasonably simple .NET 8 console application that iterates over all the packages in my au-packages repository, and then calls the PowerShell update.ps1 script in each to see if there is a new version to generate and publish.

Rather than just call out to the operating system to run each update.ps1 script, I decided to embed PowerShell in the application. This gives me a bit more control over how the scripts are run and the ability to capture any script output (and errors) from each run.

Hosting PowerShell

Figuring out how to host PowerShell in a .NET 8 application took a little bit of research. Many of the articles you find (and even some of the official documentation) are still aimed at Windows PowerShell.

The key was to reference these three NuGet packages (and use the same version of each package):

Microsoft.PowerShell.Commands.Diagnostics
Microsoft.PowerShell.SDK
System.Management.Automation

You can then create a PowerShell Class instance like this:

var iss = InitialSessionState.CreateDefault2();
iss.ExecutionPolicy = Microsoft.PowerShell.ExecutionPolicy.RemoteSigned;

var ps = PowerShell.Create(iss);

You can capture any output via the Streams property. eg. Here I am logging any errors from PowerShell as a GitHub Action error:

ps.Streams.Error.DataAdded += (_, args) =>
{
    core.WriteError(ps.Streams.Error[args.Index].ToString());
};

Running specific PowerShell cmdlets can be done via the AddCommand method. eg.

ps.AddCommand("Set-Location").AddParameter("Path", directory).Invoke();

Whereas running arbitrary PowerShell scripts is done via the AddScript method. eg.

ps.AddScript("$ErrorView = 'DetailedView'").Invoke();

If the script is in a separate .ps1 file, the only way I've found so far is to load that file into a string and pass it in. It would be nicer if you could point it at the file (so debugging/errors could include line numbers) but I have yet to find a way to do that.

var output = ps.AddScript(File.ReadAllText(Path.Combine(directory, "update.ps1")))
.Invoke();

One thing to remember is you must call the Invoke method to actually run the scripts or commands you've just added.

GitHub Action logging and summary

Because I know the application will be run in a GitHub Actions workflow, I made use of the excellent GitHub.Actions.Core NuGet package for formatting output, as well as generating a nice build summary that lists all packages that were updated in the current run.

Commit and publish

If a new package is created (eg. a .nupkg file now exists) then we assume this file can be submitted to the Chocolatey Community Repository. choco push is then called to upload the package. Remember this was where we hit that error with the trailing space? Pleasingly the .NET version doesn't exhibit this behaviour, so that problem is solved.

Assuming the package is submitted successfully then we call `git`` to stage any modified files from this package and add a tag indicating the package that was updated.

After all packages have been processed, we will commit all staged files and push the commit back to the repo, so that we get a version history of all the package changes.

Enhancements

I am currently using my fork of the chocolatey-au module, which has one minor enhancement. It adds a Files collection property to the AUPackage PowerShell class. This collection is populated with the paths of all the files that were downloaded (and had their checksums calculated).

I make use of this for some of my packages to pre-emptively upload the files to VirusTotal. This can help fast-track the packages being approved by Chocolatey as it means the Chocolatey virus scanning step is already completed. Because I use the VirusTotal CLI tool for this, it also means I can upload files up to 650MB (compared to Chocolatey's current 200MB limit due to using an older API).

I have submitted the Files property enhancement to chocolatey-au.

Summary

You can see this in action in the latest workflow runs at https://github.com/flcdrg/au-packages/actions.

Converting a Vagrant VirtualBox .box file to Hyper-V

2023-10-28T13:30:00.000+10:30

I maintain quite a number of Chocolatey packages, and sometimes I need to test a new package out, or resolve an issue that has been reported with an updated version of a package. If it is for software that I use regularly, I'll likely do the testing directly on my own machine. But if otherwise a virtual machine makes much more sense, as I can dispose of it once I'm done.

Chocolatey even provides a semi-automated way to do this using the HashiCorp tool Vagrant. They have a pre-built Windows image that is identical to the one they use for their own package verification process. Have a look at the https://github.com/chocolatey-community/chocolatey-test-environment repo to find out more.

One issue I've encountered relates to the recent upgrading of the image to version 3.0.0, which is now is based on Windows Server 2019. The previous image version (2.0.0) was built using 2012R2. Unfortunately, for some reason while the older image was provided in both VirtualBox and Hyper-V formats, the 3.0.0 image currently only has VirtualBox support. Given the choice, I'd prefer to stick with Hyper-V (rather than having to install another hypervisor on my machine). The problem is if I follow the instructions and use the default Vagrantfile from the Chocolatey test environment repository, if I only have Hyper-V installed, it will download the older 2.0.0 image. How can I use the newer one? What follows are the steps I used to create a Hyper-V compatible box file from the VirtualBox one.

First off, download the 3.0.0 image that targets VirtualBox. I don't have VirtualBox installed, but you can still tell Vagrant to download that format by providing the --provider parameter. e.g.

vagrant box add chocolatey/test-environment --provider VirtualBox

You'll see the following output (it may take a few minutes as like most Windows VM images, it is quite large)

==> box: Loading metadata for box 'chocolatey/test-environment'
    box: URL: https://vagrantcloud.com/chocolatey/test-environment
==> box: Adding box 'chocolatey/test-environment' (v3.0.0) for provider: VirtualBox
    box: Downloading: https://vagrantcloud.com/chocolatey/boxes/test-environment/versions/3.0.0/providers/VirtualBox/unknown/vagrant.box
    box:
    box: Calculating and comparing box checksum...
==> box: Successfully added box 'chocolatey/test-environment' (v3.0.0) for 'VirtualBox'!

The files for this image are saved under the vagrant.d directory in your user profile. eg. for me they're in

C:\Users\david\.vagrant.d\boxes\chocolatey-VAGRANTSLASH-test-environment\3.0.0\VirtualBox

In this directory, you can see the following files:

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a---        15/10/2023  12:38 PM           9047   box.ovf
-a---        15/10/2023  12:38 PM     7993072640 󰋊  chocolatey-test-environment-disk001.vmdk
-a---        15/10/2023  12:38 PM             26   metadata.json
-a---        15/10/2023  12:38 PM           3700   Vagrantfile

Ah haa. A .VMDK file! Now we need to convert this to a VHD. There's a few different ways to do this. The most reliable I've found is the StarWind V2V Converter tool

Now we can create a temporary virtual machine. Note that we stick with a Generation 1 VM (I tried Generation 2 and it didn't work). Also, to keep file sizes down, I stuck with a .vhd file (not a .vhdx). A .vhdx file will work but they

In an elevated prompt, run the following:

New-VM -name "test-environment2019" -VHDPath C:\tmp\chocolatey-test-environment-disk001.vhd -Generation 1

and you should see this output:

Name                 State CPUUsage(%) MemoryAssigned(M) Uptime   Status             Version
----                 ----- ----------- ----------------- ------   ------             -------
test-environment2019 Off   0           0                 00:00:00 Operating normally 11.0

Set-VMProcessor -VMName test-environment2019 -Count 4
Set-VM -VMName test-environment2019 -AutomaticCheckpointsEnabled $false -CheckpointType Disabled -AutomaticStopAction ShutDown

Start the VM and wait for it to boot. Then sign in (the password is 'vagrant')

Go to Settings, Apps and click on Oracle VM VirtualBox Guest Additions. Click Uninstall. Then click Yes to confirm. The VM will reboot.

Once it has rebooted, you can shut down the VM.

Run compress just for good measure

Optimize-VHD -Path C:\tmp\chocolatey-test-environment-disk001.vhd -Mode Full

We're now following the steps outlined in the Vagrant documentation for creating a Hyper-V base box.

Export the VM

Export-VM -VMName test-environment2019 -path c:\tmp\v3

Go to c:\tmp\v3 and delete the Snapshots folder (it's probably empty anyway)

Create a metadata.json file

I took a look at the same file in the 2.0.0 box, and it turns out this is all it contains:

{
  "provider": "hyperv"
}

Add that to the metadata.json file.

Now we need to create a .tar file (this may take a few minutes). Tar has been included with Windows since late 2017, but you could also use 7-Zip or similar.

cd C:\tmp\v3\test-environment2019\

tar cvzf c:\tmp\test-environment2019.tar ./*

Now we can add this to Vagrant using

vagrant box add C:\tmp\test-environment2019.tar --provider hyperv --name chocolatey/test-environment

You'll see output similar to the following:

==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'chocolatey/test-environment' (v0) for provider: hyperv
    box: Unpacking necessary files from: file:///C:/tmp/test-environment2019.tar
    box:
==> box: Successfully added box 'chocolatey/test-environment' (v0) for 'hyperv'!

The one issue with this is that you can see the version number of this box is v0. I'd much prefer to set the version 3.0.0. It turns out you can't set that directly via the command line, but there is a workaround.

Create another metadata.json file (in c:\tmp) and add the following content:

{
  "name": "chocolatey/test-environment",
  "versions": [
    {
      "version": "3.0.0",
      "status": "active",
      "providers": [
        {
          "name": "hyperv",
          "url": "file:///C:/tmp/test-environment2019.tar"
        }
      ]
    }
  ]
}

and add the new box to Vagrant using this file instead:

vagrant box add .\metadata.json

And now we see this output

==> box: Loading metadata for box '.\metadata.json'
    box: URL: file://C:/tmp/metadata.json
==> box: Adding box 'chocolatey/test-environment' (v3.0.0) for provider: hyperv
    box: Unpacking necessary files from: file:///C:/tmp/test-environment2019.tar
    box:
==> box: Successfully added box 'chocolatey/test-environment' (v3.0.0) for 'hyperv'!

We can now list all the boxes Vagrant knows about:

vagrant box list

chocolatey/test-environment (hyperv, 0)
chocolatey/test-environment (hyperv, 2.0.0)
chocolatey/test-environment (hyperv, 3.0.0)
chocolatey/test-environment (VirtualBox, 3.0.0)

I just want the 'hyperv 3.0.0' box, so I'll remove the others

vagrant box remove chocolatey/test-environment --box-version 0 --provider hyperv
vagrant box remove chocolatey/test-environment --box-version 2.0.0 --provider hyperv
vagrant box remove chocolatey/test-environment --box-version 3.0.0 --provider VirtualBox

And now you should be fine to run vagrant up to provision a new VM using the Hyper-V provider, and it will use Windows Server 2019!

Here's an example of doing this with the chocolatey-test-environment (run from an elevated prompt):

vagrant up

Which gives the following output (including signing in with your local username and password):

Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
    default: Configuring the VM...
    default: Setting VM Integration Services
==> default: guest_service_interface is enabled
==> default: heartbeat is enabled
==> default: key_value_pair_exchange is enabled
==> default: shutdown is enabled
==> default: time_synchronization is enabled
==> default: vss is enabled
    default: Setting VM Enhanced session transport type to disabled/default (VMBus)

Vagrant requires administrator access for pruning SMB shares and
may request access to complete removal of stale shares.
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
    default: Timeout: 130 seconds
    default: IP: 172.20.15.213
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 172.20.15.213:5985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
    default: You will be asked for the username and password to use for the SMB
    default: folders shortly. Please use the proper username/password of your
    default: account.
    default:
    default: Username (user[@domain]): david
    default: Password (will be hidden):

Vagrant requires administrator access to create SMB shares and
may request access to complete setup of configured shares.
==> default: Mounting SMB shared folders...
    default: C:/dev/git/chocolatey-test-environment/packages => /packages
    default: C:/dev/git/chocolatey-test-environment => /vagrant
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

Windows Hyper-V Manager will show the new VM running:

You can also connect to the VM and sign in to confirm it is running Windows Server 2019 and working as expected:

Vagrant error

When you run vagrant up, it fails with the following error (observed with Vagrant 2.3.7):

An error occurred executing a remote WinRM command.

Shell: Cmd
Command: hostname
Message: Digest initialization failed: initialization error

Apparently this problem is solved with Vagrant version 2.3.8. Ensure you're you're using Vagrant 2.3.8 or newer.

Choco list -localonly (Feb 2019 Edition)

2019-02-03T21:15:00.000+10:30

What software / applications am I using on my laptop (February 2019 Edition) according to Chocolatey? Here's an edited list of the output from choco list -localonly:

Package	Version	Comments
7zip	18.6
7zip.commandline	16.02.0.20170209
audacity	2.3.0	Audio editor
becyicongrabber	2.30.0.20161027	Icon extractor (for creating Chocolatey packages)
beyondcompare	4.2.9.23626	My favourite file comparison tool
beyondcompare-integration	1.0.1	Configure Beyond Compare for TortoiseGit/Svn
dellcommandupdate-uwp	3.0.0	Dell's driver update app
dns-benchmark	1.3.6668.0	Useful DNS checker
docker-for-windows	99.0.0.0
dotnetdeveloperbundle	2.3.0.2563	RedGate's .NET tools
dropbox	41.4.80
fiddler	5.0.20182.28034
FiraCode	1.206	Nice developer font
Firefox	57.0.4
git	2.20.1
GoogleChrome	63.0.3239.132
iTunes	12.9.3.3
keepass	2.41	Password manager
microsoft-teams	1.1.00.2251
mousewithoutborders	2.1.8.105	Share mouse across laptop and desktop PCs
msbuild-structured-log-viewer	2.0.61
nodejs	11.9.0
notepadplusplus	7.6.3	Using this less now compared to VS Code
nuget.commandline	4.9.2
obs-studio	22.0.2	Video / screen recording
OctopusTools	5.2.0
Office365ProPlus	2016.20170321
paint.net	4.1.5
PDFXchangeEditor	7.0.328.2	My favourite
Pester	4.4.1	PowerShell unit tests
pingplotter	5.8.10	Useful visual ping network status
powershell-core	6.1.2
procmon	3.50	SysInternals Process Monitor
resharper-clt.portable	2018.3.2	ReSharper's free command-line tools
resharper-ultimate-all	2018.3.2
screentogif	2.16	Handy screen recorder
skype	8.38.0.138
snagit	2019.1.0	Screen grabber
sql-server-2017	14.0.1000
sql-server-management-studio	14.0.17289.1
tailblazer	0.9.0.536	Text/log file viewer
tortoisegit	2.7.0.0
tortoisesvn	1.11.1.28492
ubiquiti-unifi-controller	5.9.29	Software for managing UniFi wireless access points
vagrant	2.2.3	Manage virtual machines
visualstudio2017enterprise	15.2.26430.20170605
visualstudiocode	1.19.3
vsts-cli	0.1.4.20190126	Command line tool for managing Azure DevOps
vswhere	2.6.7
wifiinfoview	2.42
windirstat	1.1.2.20161210	Where's all that disk space being used?
wireshark	2.6.5
x-lite	5.4.0.94388	VoIP client
yarn	1.13.0
zoomit	4.50.0.20160210	Great for presentations

This is also a good basis for refreshing my Boxstarter scripts.

Boxstarter and Chocolatey tips

2018-07-16T10:00:00.000+09:30

Two big things happened earlier this year on the Chocolatey front. First off, Boxstarter (the tool created by Matt Wrock that allows you to script up full Windows installations including handling reboots) is now being managed by Chocolatey. Boxstarter.org still exists, but the source repository is now under the Chocolatey org on GitHub.

The second is that Microsoft are contributing Boxstarter scripts in a new GitHub repo – https://github.com/Microsoft/windows-dev-box-setup-scripts

If you’re looking to use Boxstarter to automate the software installation of your Windows machines, there’s a few tricks and traps worth knowing about.

Avoid MAXPATH errors

It’s worth understanding that Boxstarter embeds its own copy of Chocolatey and uses that rather than choco.exe. Due to some compatibility issues Boxstarter currently needs to embed an older version of Chocolatey. That particular version does have one known bug where the temp directory Chocolatey uses to download binaries goes one directory deeper each install. Not a problem in isolation, but when you’re installing a lot of packages all at once, you soon hit the old Windows MAXPATH limit. A workaround is described in the bug report – essentially using the --cache-location argument to override where downloads are saved. The trick here is that you need to use this on all choco calls in your Boxstarter script – even for things like choco pin. Forget those and you still may experience the MAXPATH problem.

To make it easier, I add the following lines to the top of my Boxstarter scripts

New-Item -Path "$env:userprofile\AppData\Local\ChocoCache" -ItemType directory -Force | Out-Null
$common = "--cacheLocation=`"$env:userprofile\AppData\Local\ChocoCache`""

And then I can just append $common to each choco statement. eg.

cinst nodejs $common
cinst visualstudiocode $common
choco pin add -n=visualstudiocode $common

Avoid unexpected reboots

Detecting and handling reboots is one of the great things about Boxstarter. You can read more in the docs, but one thing to keep in mind is it isn’t perfect. If a reboot is initiated without Boxstarter being aware of it, then it can’t do its thing to restart and continue.

One command I’ve found that can cause this is using Enable-WindowsOptionalFeature. If the feature you’re turning on needs a restart, then Boxstarter won’t resume afterwards. The workaround here is to leverage Chocolatey’s support for the windowsfeatures source. So instead of this

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

Do this

choco install Microsoft-Hyper-V-All -source windowsfeatures $common

Logging

If you have a more intricate Boxstarter script, you may run into some problems that you need to diagnose. Don’t look in the usual Chocolatey.log as you won’t see anything there. Boxstarter logs all output to its own log, which by default ends up in $env:LocalAppData\Boxstarter\Boxstarter.log. This becomes even more useful when you consider that Boxstarter may automatically restart your machine multiple times, so having a persistent record of what happened is invaluable. The other things you might want to make use of is Boxstarter-specific commands like Write-BoxstarterMessage (which writes to the log file as well as the console output) and Log-BoxstarterMessage (which just write to the log file)

Find out more about these and other logging commands by running help about_boxstarter_logging.

My scripts

I keep a few of my Boxstarter scripts at https://gist.github.com/flcdrg/87802af4c92527eb8a30. Feel free to have a look and borrow them if they look useful.

Find out more

If you’re really getting in to Chocolatey and Boxstarter, you might also be interested in Chocolatey Fest, a conference focusing on Windows automation being held San Francisco on October 8th.