• Get a list of Azure Pipelines and YAML files

    I wanted to document the pipelines in a particular Azure DevOps project. Rather than manually write up the name of each pipeline and the corresponding YAML file, I figured there must be a way to query that data.

    I’ve done something similar in the past using the Azure DevOps REST API, but this time I’m using the Azure CLI.

    Make sure you have the devops extension installed (az extension add --name azure-devops if you don’t have it already). The commands provided by this extension use the same REST API under the hood that we used directly last time.

    I can get a list of pipelines for the current project with az pipelines list.

    This command returns a list of objects corresponding to the BuildDefinitionReference data structure. While it has the pipeline name, I noticed that doesn’t include any information about the YAML file. To get that you need query an individual pipeline using:

    az pipelines show --name PipelineName
    

    This produces a BuildDefinition object, which happens to include a process property. While it isn’t documented in the BuildProcess data structure, if you look at the actual data you’ll see not only the type property but a yamlFilename property, which is just what we want.

    "process": {
        "type": 2,
        "yamlFilename": "release.yaml"
      }
    

    Putting it all together, and taking advantage of the JMESPath query to limit which fields we get back, I can produce a comma-separated list of pipeline names and their corresponding YAML files with the following:

    (az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename]" | ConvertFrom-Json) -join "," }
    

    So for this project:

    Screenshot of Azure Pipelines in a project

    You get this:

    Alternate,alternate.yml
    ReleaseTest,release.yaml
    

    This will be more useful in a project with many more pipelines. If the project has multiple repositories you could also include the repository name as well.

    e.g.

    (az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename, repository.name]" | ConvertFrom-Json) -join "," }
    

    Such a project would produce something similar to this:

    Custom Git,azure-pipelines.yml,Repro
    Repro,azure-pipelines.yml,Repro
    task-test,azure-pipelines.yml,task-test
    

    You can include extra columns of data as needed.

  • 2 years at SixPivot

    The 1st of February means I tick over two years at SixPivot! I’m pleased to report things continue to go well.

    Take a look back at my thoughts on joining SixPivot, and my first year anniversary post.

    SixPivot’s presence in Adelaide has now grown to 5. We all work from home, but I think we might have a few local catchups this year. (Head over to the SixPivot team page to see our full list of staff)

    It’s a great place to work, and I’m looking forward to what 2023 brings.

  • Provision an Azure Virtual Machine with Terraform Cloud

    Terraform logo

    Sometimes I need to spin up a virtual machine to quickly test something out on a ‘vanilla’ machine, for example, to test out a Chocolatey package that I maintain.

    Most of the time I log in to the Azure Portal and click around to create a VM. The addition of the choice to use the Azure Virtual Machine with preset configuration does make it a bit easier, but it’s still lots of clicking. Maybe I should try automating this!

    There are a few choices for automating, but seeing as I’ve been using Terraform lately I thought I’d try that out, together with Terraform Cloud. As I’ll be putting the Terraform files in a public repository on GitHub, I can use the free tier for Terraform Cloud.

    You can find the source for the Terraform files at https://github.com/flcdrg/terraform-azure-vm/.

    You’ll also need to have both the Azure CLI and Terraform CLI installed. You can do this easily via Chocolatey:

    choco install terraform
    choco install azure-cli
    

    Setting up Terraform Cloud Workspace with GitHub

    1. Log in (or sign up) to Terraform Cloud at https://app.terraform.io, select (or create) your organisation, then go to Workspaces and click on Create a workspace Terraform Cloud - Workspaces tab
    2. Select how you’d like to trigger a workflow. To keep things simple, I chose Version control workflow Terraform Cloud - Create a new workspace
    3. Select the version control provider - Github.com. Alt text
    4. You will now need to authenticate with GitHub. Alt text
    5. Watch out if you get a notification about a pop-up blocker. Alt text If you do, then enable pop-ups for this domain Alt text
    6. Choose which GitHub account or organisation to use: Alt text
    7. Select which repositories should be linked to Terraform Cloud. Alt text
    8. If you use multi-factor authentication then you’ll need to approve the access. Alt text
    9. Now that your GitHub repositories are connected, you need to select the repository that Terraform Cloud will use for this workspace. Alt text
    10. Enter a workspace name (and optionally a description) Alt text
    11. Now your workspace has been created! Alt text

    You’re now ready to add Terraform files to your GitHub repository. I like to use the Terraform CLI to validate and format my .tf files before I commit them to version control.

    After adding versions.tf file that contains a cloud definition (along with any providers), you can run terraform login

    terraform {
      cloud {
        organization = "flcdrg"
        hostname     = "app.terraform.io"
    
        workspaces {
          name = "terraform-azure-vm"
        }
      }
    
      required_providers {
        azurerm = {
          source  = "hashicorp/azurerm"
          version = "=3.39.1"
        }
        random = {
          source  = "hashicorp/random"
          version = "3.4.3"
        }
      }
    }
    
    provider "azurerm" {
      features {}
    }
    

    A browser window will launch to allow you to create an API token that you can then paste back into the CLI. Terraform Cloud - Create API Token dialog

    The next thing we need to do is create an Azure service principal that Terraform Cloud can use when deploying to Azure.

    In my case, I created a resource group and granted the service principal Contributor access to it (assuming that all the resources you want Terraform to create will live within that resource group). You could also allow the service principal access to the whole subscription if you prefer.

    az login
    az group create --location westus --resource-group MyResourceGroup
    az ad sp create-for-rbac --name <service_principal_name> --role Contributor --scopes /subscriptions/<subscription_id>/resourceGroups/<resourceGroupName>
    

    Now go back to Terraform Cloud, and after selecting the newly created workspace, select Variables.

    Under Workspace variables, click Add variable, then select Environment variables. Add a variable for each of the following (for ARM_CLIENT_SECRET also check the Sensitive checkbox), for the value copy the appropriate value from the output from creating the service principal:

    • ARM_CLIENT_ID - appId
    • ARM_CLIENT_SECRET - password
    • ARM_SUBSCRIPTION_ID - id from az account show
    • ARM_TENANT_ID

    Workspace variables

    With those variables set, you can now push your Terraform files to the GitHub repository.

    The Terraform Cloud workspace is configured to evaluate a plan on pull requests, and on pushes or merges to main it will apply those changes.

    Terraform Cloud - Plan

    By default, you need to manually confirm before ‘apply’ will run (you can change the workspace to auto-approve to avoid this). Confirm Plan dialog

    After a short wait, all the Azure resources (including the VM) should have been created and ready to use. Terraform Cloud - changes applied

    Virtual machine password

    I’m not hardcoding the password for the virtual machine - rather I’m using the Terraform random_password resource to generate a random password. The password is not displayed in the logs as it is marked as ‘sensitive’. But I will actually need to know the password so I can RDP to the VM. It turns out the password value is saved in Terraform state, and you can examine this via the States tab of the workspace.

    Terraform Cloud - Workspace state

    With that, I’m now able to navigate to the VM resource in the Azure Portal and connect via RDP and do what I need to do.

    If you wanted to stick with the CLI, you can also use Azure PowerShell to launch an RDP session.

    Extra configuration

    If you review the Terraform in the repo, you’ll notice I also make use of the azurerm_virtual_machine_extension resource to run some PowerShell that installs Chocolatey. That just saves me from having to do it manually. If you can automate it, why not!

    Cleaning up when you’re done

    For safety, the virtual machine is set to auto shutdown in the evening, which will reduce any costs. To completely remove the virtual machine and any associated storage you can run a “destroy plan”

    From the workspace, go to Settings, then Destruction and deletion, and click Queue destroy plan.

    Alt text