Converting a Vagrant VirtualBox .box file to Hyper-V

2023-10-28T13:30:00.000+10:30

I maintain quite a number of Chocolatey packages, and sometimes I need to test a new package out, or resolve an issue that has been reported with an updated version of a package. If it is for software that I use regularly, I'll likely do the testing directly on my own machine. But if otherwise a virtual machine makes much more sense, as I can dispose of it once I'm done.

Chocolatey even provides a semi-automated way to do this using the HashiCorp tool Vagrant. They have a pre-built Windows image that is identical to the one they use for their own package verification process. Have a look at the https://github.com/chocolatey-community/chocolatey-test-environment repo to find out more.

One issue I've encountered relates to the recent upgrading of the image to version 3.0.0, which is now is based on Windows Server 2019. The previous image version (2.0.0) was built using 2012R2. Unfortunately, for some reason while the older image was provided in both VirtualBox and Hyper-V formats, the 3.0.0 image currently only has VirtualBox support. Given the choice, I'd prefer to stick with Hyper-V (rather than having to install another hypervisor on my machine). The problem is if I follow the instructions and use the default Vagrantfile from the Chocolatey test environment repository, if I only have Hyper-V installed, it will download the older 2.0.0 image. How can I use the newer one? What follows are the steps I used to create a Hyper-V compatible box file from the VirtualBox one.

First off, download the 3.0.0 image that targets VirtualBox. I don't have VirtualBox installed, but you can still tell Vagrant to download that format by providing the --provider parameter. e.g.

vagrant box add chocolatey/test-environment --provider VirtualBox

You'll see the following output (it may take a few minutes as like most Windows VM images, it is quite large)

==> box: Loading metadata for box 'chocolatey/test-environment'
    box: URL: https://vagrantcloud.com/chocolatey/test-environment
==> box: Adding box 'chocolatey/test-environment' (v3.0.0) for provider: VirtualBox
    box: Downloading: https://vagrantcloud.com/chocolatey/boxes/test-environment/versions/3.0.0/providers/VirtualBox/unknown/vagrant.box
    box:
    box: Calculating and comparing box checksum...
==> box: Successfully added box 'chocolatey/test-environment' (v3.0.0) for 'VirtualBox'!

The files for this image are saved under the vagrant.d directory in your user profile. eg. for me they're in

C:\Users\david\.vagrant.d\boxes\chocolatey-VAGRANTSLASH-test-environment\3.0.0\VirtualBox

In this directory, you can see the following files:

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a---        15/10/2023  12:38 PM           9047   box.ovf
-a---        15/10/2023  12:38 PM     7993072640 󰋊  chocolatey-test-environment-disk001.vmdk
-a---        15/10/2023  12:38 PM             26   metadata.json
-a---        15/10/2023  12:38 PM           3700   Vagrantfile

Ah haa. A .VMDK file! Now we need to convert this to a VHD. There's a few different ways to do this. The most reliable I've found is the StarWind V2V Converter tool

Now we can create a temporary virtual machine. Note that we stick with a Generation 1 VM (I tried Generation 2 and it didn't work). Also, to keep file sizes down, I stuck with a .vhd file (not a .vhdx). A .vhdx file will work but they

In an elevated prompt, run the following:

New-VM -name "test-environment2019" -VHDPath C:\tmp\chocolatey-test-environment-disk001.vhd -Generation 1

and you should see this output:

Name                 State CPUUsage(%) MemoryAssigned(M) Uptime   Status             Version
----                 ----- ----------- ----------------- ------   ------             -------
test-environment2019 Off   0           0                 00:00:00 Operating normally 11.0

Set-VMProcessor -VMName test-environment2019 -Count 4
Set-VM -VMName test-environment2019 -AutomaticCheckpointsEnabled $false -CheckpointType Disabled -AutomaticStopAction ShutDown

Start the VM and wait for it to boot. Then sign in (the password is 'vagrant')

Go to Settings, Apps and click on Oracle VM VirtualBox Guest Additions. Click Uninstall. Then click Yes to confirm. The VM will reboot.

Once it has rebooted, you can shut down the VM.

Run compress just for good measure

Optimize-VHD -Path C:\tmp\chocolatey-test-environment-disk001.vhd -Mode Full

We're now following the steps outlined in the Vagrant documentation for creating a Hyper-V base box.

Export the VM

Export-VM -VMName test-environment2019 -path c:\tmp\v3

Go to c:\tmp\v3 and delete the Snapshots folder (it's probably empty anyway)

Create a metadata.json file

I took a look at the same file in the 2.0.0 box, and it turns out this is all it contains:

{
  "provider": "hyperv"
}

Add that to the metadata.json file.

Now we need to create a .tar file (this may take a few minutes). Tar has been included with Windows since late 2017, but you could also use 7-Zip or similar.

cd C:\tmp\v3\test-environment2019\

tar cvzf c:\tmp\test-environment2019.tar ./*

Now we can add this to Vagrant using

vagrant box add C:\tmp\test-environment2019.tar --provider hyperv --name chocolatey/test-environment

You'll see output similar to the following:

==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'chocolatey/test-environment' (v0) for provider: hyperv
    box: Unpacking necessary files from: file:///C:/tmp/test-environment2019.tar
    box:
==> box: Successfully added box 'chocolatey/test-environment' (v0) for 'hyperv'!

The one issue with this is that you can see the version number of this box is v0. I'd much prefer to set the version 3.0.0. It turns out you can't set that directly via the command line, but there is a workaround.

Create another metadata.json file (in c:\tmp) and add the following content:

{
  "name": "chocolatey/test-environment",
  "versions": [
    {
      "version": "3.0.0",
      "status": "active",
      "providers": [
        {
          "name": "hyperv",
          "url": "file:///C:/tmp/test-environment2019.tar"
        }
      ]
    }
  ]
}

and add the new box to Vagrant using this file instead:

vagrant box add .\metadata.json

And now we see this output

==> box: Loading metadata for box '.\metadata.json'
    box: URL: file://C:/tmp/metadata.json
==> box: Adding box 'chocolatey/test-environment' (v3.0.0) for provider: hyperv
    box: Unpacking necessary files from: file:///C:/tmp/test-environment2019.tar
    box:
==> box: Successfully added box 'chocolatey/test-environment' (v3.0.0) for 'hyperv'!

We can now list all the boxes Vagrant knows about:

vagrant box list

chocolatey/test-environment (hyperv, 0)
chocolatey/test-environment (hyperv, 2.0.0)
chocolatey/test-environment (hyperv, 3.0.0)
chocolatey/test-environment (VirtualBox, 3.0.0)

I just want the 'hyperv 3.0.0' box, so I'll remove the others

vagrant box remove chocolatey/test-environment --box-version 0 --provider hyperv
vagrant box remove chocolatey/test-environment --box-version 2.0.0 --provider hyperv
vagrant box remove chocolatey/test-environment --box-version 3.0.0 --provider VirtualBox

And now you should be fine to run vagrant up to provision a new VM using the Hyper-V provider, and it will use Windows Server 2019!

Here's an example of doing this with the chocolatey-test-environment (run from an elevated prompt):

vagrant up

Which gives the following output (including signing in with your local username and password):

Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
    default: Configuring the VM...
    default: Setting VM Integration Services
==> default: guest_service_interface is enabled
==> default: heartbeat is enabled
==> default: key_value_pair_exchange is enabled
==> default: shutdown is enabled
==> default: time_synchronization is enabled
==> default: vss is enabled
    default: Setting VM Enhanced session transport type to disabled/default (VMBus)

Vagrant requires administrator access for pruning SMB shares and
may request access to complete removal of stale shares.
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
    default: Timeout: 130 seconds
    default: IP: 172.20.15.213
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 172.20.15.213:5985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
    default: You will be asked for the username and password to use for the SMB
    default: folders shortly. Please use the proper username/password of your
    default: account.
    default:
    default: Username (user[@domain]): david
    default: Password (will be hidden):

Vagrant requires administrator access to create SMB shares and
may request access to complete setup of configured shares.
==> default: Mounting SMB shared folders...
    default: C:/dev/git/chocolatey-test-environment/packages => /packages
    default: C:/dev/git/chocolatey-test-environment => /vagrant
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

Windows Hyper-V Manager will show the new VM running:

You can also connect to the VM and sign in to confirm it is running Windows Server 2019 and working as expected:

Vagrant error

When you run vagrant up, it fails with the following error (observed with Vagrant 2.3.7):

An error occurred executing a remote WinRM command.

Shell: Cmd
Command: hostname
Message: Digest initialization failed: initialization error

Apparently this problem is solved with Vagrant version 2.3.8. Ensure you're you're using Vagrant 2.3.8 or newer.

Downloading an Azure VM

2020-04-03T09:45:00.000+10:30

Yesterday I needed to get a copy of a virtual machine onto my local workstation. As I'm now working from home, that was going to mean downloading a lot of data, but first I had to find the VM. I remembered I had exported this particular VM up into Azure at one stage to experiment with using different hardware specs to find out how that would affect performance.

Lucky for me, the VM was still there (though de-allocated to reduce costs). Usually you want to migrate a VM up into the cloud, but I needed to go the other way! So how do you get a copy of that VM? It turns out it isn't that tricky:

Make sure the VM is shut down (mine was)
Open up the VM in the Azure Portal
Under Settings, click on Disks
Click on the individual disk (if you have more than one, you'll need to repeat the next few steps)
Under Settings, click on Disk Export
You're prompted to enter a URL expire time. The default is 3600 seconds (1 hour). If you have limited bandwidth you should make this larger, otherwise your download may fail. I set mine to 36000 (10 hours)
Click Generate URL and a URL will be displayed
Download the .vhd file for this disk. Mine was 80GB and it took all day. It also failed a number of times, but I was able to restart the download and it did continue on from where it left off.
The download defaulted to calling the file abcd, but it is a VHD file, so just rename the file to something useful.

David Gardiner - Virtual Machines

Converting a Vagrant VirtualBox .box file to Hyper-V

Vagrant error

Downloading an Azure VM