Your server version will lose standard Azure support on March 31, 2026. Upgrade now to avoid extended support charges starting April 1, 2026.

Terraform was being used for Infrastructure as Code, and it looked similar to this:

resource "azurerm_postgresql_flexible_server" "server" {
  name                              = "psql-postgresql-apps-australiaeast"
  resource_group_name               = data.azurerm_resource_group.rg.name
  location                          = data.azurerm_resource_group.rg.location
  version                           = "11"
  delegated_subnet_id               = azurerm_subnet.example.id
  private_dns_zone_id               = azurerm_private_dns_zone.example.id
  public_network_access_enabled     = false
  administrator_login               = "psqladmin"
  administrator_password_wo         = ephemeral.random_password.postgresql_password.result
  administrator_password_wo_version = 1
  zone                              = "1"

  storage_mb   = 32768
  storage_tier = "P4"

  sku_name   = "B_Standard_B1ms"
  depends_on = [azurerm_private_dns_zone_virtual_network_link.example]
}

As you can see from the code and screenshot above, the PostgreSQL version in use was 11. Doing a bit of research, I found version 11 was first released back in 2018, and the the final minor update 11.22 was released in 2023.

Azure provides standard support for PostgreSQL versions (documented at Azure Database for PostgreSQL version policy). There is also the option of paying for extended support, though in the case of v11 that only gets you to November this year, so just a few extra months.

In my case, I wanted to do a test of the upgrade process first, so I restored a backup of the existing server to a new resource. This essentially creates an exact copy of the server at the same version.

While we are using Infrastructure as Code, I decided to use the Azure Portal to test the upgrade, as I figured if there were any problems, they might be easier to understand, rather than try and interpret weird Terraform/AzureRM errors.

Following the upgrade documentation, I clicked on the Upgrade in the Portal.

This initiates a deployment, which depending on how much data you have and the particular SKU you're running on (eg. how fast the VM you're using is), this may take quite a while. One time it took over an hour (which was important as that may be longer than the default Terraform lifecycle, and also the pipeline job timeouts).

If that succeeds, then you should be good to try the real thing with IaC.

Upgrading with Terraform

To upgrade a major version with Terraform, you need to make a couple of changes to your azurerm_postgresql_flexible_server resource:

1. The version property should be updated to the desired version
2. The create_mode property should be set to Update (if it wasn't specified then the default is 'Default')

resource "azurerm_postgresql_flexible_server" "server" {
  name                              = "psql-postgresql-apps-australiaeast"
  resource_group_name               = data.azurerm_resource_group.rg.name
  location                          = data.azurerm_resource_group.rg.location
  version                           = "17"
  delegated_subnet_id               = azurerm_subnet.example.id
  private_dns_zone_id               = azurerm_private_dns_zone.example.id
  public_network_access_enabled     = false
  administrator_login               = "psqladmin"
  administrator_password_wo         = ephemeral.random_password.postgresql_password.result
  administrator_password_wo_version = 1
  zone                              = "1"
  create_mode                       = "Update"

  storage_mb   = 32768
  storage_tier = "P4"

  sku_name   = "B_Standard_B1ms"
  depends_on = [azurerm_private_dns_zone_virtual_network_link.example]
}

The weird thing (which I assume is a side-effect of Terraform state) is that even after you've completed the upgrade, you can't change create_mode back to Default - Terraform will throw an error if you try that. Instead you just need to leave it set to Update, but as long as the version property doesn't change then Terraform will leave it at the same version.

Adjust your timeouts

I was using Azure Pipelines, so I added a timeoutInMinutes property to the job and set it to 90 minutes. Be aware that there are different default and maximum timeouts depending on what kind of build agent you use.

Likewise the Terraform azurerm_postgresql_flexible_server resource has default timeouts. You may want to specify a timeout block to extend those values if necessary.

Gotchas

I hit some compatibility issues with the PostgreSQL instance I was attempting to upgrade. The Portal displayed the following error(s):

The major version upgrade failed precheck. Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading with password authentication mode enabled is not allowed from source version MajorVersion11. Please enable SCRAM and reset the passwords prior to retrying the upgrade.

There's two issues here:

• The pg_failover_slots shared preloaded library is not supported for upgrading
• Legacy MD5 passwords are deprecated in newer versions, and "SCRAM" needs to be enabled

How do we resolve these with Infrastructure as Code? In this case as we're using Terraform, we need to map/import those settings and then we can modify them. We make use of the azurerm_postgresql_flexible_server_configuration resource for this.

The value properties should initially match the existing values (eg. Make sure that Terraform thinks they are unchanged). A trick to get the existing values is to run the Terraform in 'plan' mode and take note of what values it can see from and then copy those into your code.

import {
  to = azurerm_postgresql_flexible_server_configuration.accepted_pasword_auth_method
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/azure.accepted_password_auth_method"
}

resource "azurerm_postgresql_flexible_server_configuration" "accepted_pasword_auth_method" {
  name      = "azure.accepted_password_auth_method"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5"
}

import {
  to = azurerm_postgresql_flexible_server_configuration.password_encryption
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/password_encryption"
}

resource "azurerm_postgresql_flexible_server_configuration" "password_encryption" {
  name      = "password_encryption"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5"
}

import {
  to = azurerm_postgresql_flexible_server_configuration.shared_preload_libraries
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/shared_preload_libraries"
}

resource "azurerm_postgresql_flexible_server_configuration" "shared_preload_libraries" {
  name      = "shared_preload_libraries"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "anon,auto_explain,pg_cron,pg_failover_slots,pg_hint_plan,pg_partman_bgw,pg_prewarm,pg_stat_statements,pgaudit,pglogical,timescaledb,wal2json"
}

Once you've got those in place then you can make the changes to remove the upgrade block:

resource "azurerm_postgresql_flexible_server_configuration" "accepted_pasword_auth_method" {
  name      = "azure.accepted_password_auth_method"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5,SCRAM-SHA-256"
}

resource "azurerm_postgresql_flexible_server_configuration" "password_encryption" {
  name      = "password_encryption"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "SCRAM-SHA-256"
}

resource "azurerm_postgresql_flexible_server_configuration" "shared_preload_libraries" {
  name      = "shared_preload_libraries"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "anon,auto_explain,pg_cron,pg_hint_plan,pg_partman_bgw,pg_prewarm,pg_stat_statements,pgaudit,pglogical,timescaledb,wal2json"
}

This will allow any existing MD5 passwords to continue to work, but any new passwords will use the more modern SCRAM-SHA-256.

For the shared_preload_libraries, we've removed the offending pg_failover_slots from the list.

Tips

• Temporarily upgrade your server SKU to beefier hardware so the upgrade goes faster. If you're using IaC then make sure you use that to make the change.
• Note that if you change the separate storage performance tier (IOPS), you will need to wait 12 hours before downgrading again.

Completion

If everything goes to plan, you should end up with your PostgreSQL resource upgraded to the version that you specified. Here's my resource upgraded to 17.7. v18 is actually available but I wasn't offered it due to 'regional capacity constraints', which explains why the 'Upgrade' button is now disabled.

I've published source code for a working example of Azure Database for PostgreSQL flexible server with an Azure Container app and using a VNet at https://github.com/flcdrg/terraform-azure-postgresql-containerapps

Sometimes when you're using Terraform for your Infrastructure as Code with Azure, it's a bit tricky to match up what you can see in the Azure Portal versus the Terraform resources in the AzureRM provider. Enabling auditing in Azure SQL is a great example.

In the Azure Portal, select your Azure SQL resource, then expand the Security menu and select Auditing. You can then choose to Enable Azure SQL Auditing, and upon doing this you can then choose to send auditing data to any or all of Azure Storage, Log Analytics and/or Event Hub.

It's also worth highlighting that usually you'd enable auditing at the server level, but it is also possible to enable it per database.

The two Terraform resources you may have encountered to manage this are mssql_server_extended_auditing_policy and mssql_database_extended_auditing_policy.

It's useful to refer back to the Azure SQL documentation on setting up auditing to understand how to use these.

A couple of points that are worth highlighting:

1. If you don't use the audit_actions_and_groups property, the default groups of actions that will be audited are:

   BATCH_COMPLETED_GROUP SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP FAILED_DATABASE_AUTHENTICATION_GROUP

2. If you do define auditing at the server level, the policy applies to all existing and newly created databases on the server. If you define auditing at the database level, the policy will apply in addition to any server level settings. So be careful you don't end up auditing the same thing twice unintentionally!

Sometimes it can also be useful to review equivalent the Bicep/ARM definitions Microsoft.Sql/servers/extendedAuditingSettings, as sometimes they can clarify how to use various properties.

You'll see both the Terraform and Bicep have properties to configure using a Storage Account, but while you can see Log Analytics and Event Hub in the Portal UI, it's not obvious how those set up.

The simplest policy you can set is this:

resource "azurerm_mssql_server_extended_auditing_policy" "auditing" {
  server_id = azurerm_mssql_server.mssql.id
}

This enables the server auditing policy, but the data isn't going anywhere yet!

Storage account

When you select an Azure Storage Account for storing auditing data, you will end up with a bunch .xel files created under a sqldbauditlogs blob container.

There are a number of ways to view the .xel files, documented here

Using a storage account for storing auditing has a few variations, depending on how you want to authenticate to the Storage Account.

Access key

resource "azurerm_mssql_server_extended_auditing_policy" "auditing" {
  server_id = azurerm_mssql_server.mssql.id

  storage_endpoint                        = azurerm_storage_account.storage.primary_blob_endpoint
  storage_account_access_key              = azurerm_storage_account.storage.primary_access_key
  storage_account_access_key_is_secondary = false
  retention_in_days                       = 6
}

Normally storage_account_access_key_is_secondary would be set to false, but if you are rotating your storage access keys, then you may choose to switch to the secondary key while you're rotating the primary.

Managed identity

You can also use managed identity to authenticate to the storage account. In this case you don't supply the access_key properties, but you will need to add a role assignment granting the Storage Blob Data Contributor role to the identity of your Azure SQL resource.

resource "azurerm_mssql_server_extended_auditing_policy" "auditing" {
  server_id = azurerm_mssql_server.mssql.id

  storage_endpoint  = azurerm_storage_account.storage.primary_blob_endpoint
  retention_in_days = 6
}

Log analytics workspaces

To send data to a Log Analytics Workspace, the log_monitoring_enabled property needs to be set to true. This is the default.

But to tell it which workspace to send the data to, you need to add a azurerm_monitor_diagnostic_setting resource.

resource "azurerm_monitor_diagnostic_setting" "mssql_server_to_log_analytics" {
  name                       = "example-diagnostic-setting"
  target_resource_id         = "${azurerm_mssql_server.mssql.id}/databases/master"
  log_analytics_workspace_id = azurerm_log_analytics_workspace.la.id

  enabled_log {
    category = "SQLSecurityAuditEvents"
  }
}

Note that for the server policy, you set the target_resource_id to the master database of the server, not the resource id of the server itself.

Here's what the auditing data looks like when viewed in Log Analytics:

Event Hub

Likewise, if you want data to go to an Event Hub, you need to use the azurerm_monitor_diagnostic_setting resource.

resource "azurerm_monitor_diagnostic_setting" "mssql_server_to_event_hub" {
  name                           = "ds_mssql_event_hub"
  target_resource_id             = "${azurerm_mssql_server.mssql.id}/databases/master"
  eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.eh.id
  eventhub_name                  = azurerm_eventhub.eh.name

  enabled_log {
    category = "SQLSecurityAuditEvents"
  }
}

Multiple destinations

As is implied by the Azure Portal, you can have one, two or all three destinations enabled for auditing. But it isn't immediately obvious that you should only have one azurerm_monitor_diagnostic_setting for your server auditing - don't create separate azurerm_monitor_diagnostic_setting resources for each destination - Azure will not allow it.

For example, if you're going to log to all three, you'd have a single diagnostic resource like this:

resource "azurerm_monitor_diagnostic_setting" "mssql_server" {
  name                           = "diagnostic_setting"
  target_resource_id             = "${azurerm_mssql_server.mssql.id}/databases/master"
  eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.eh.id
  eventhub_name                  = azurerm_eventhub.eh.name

  log_analytics_workspace_id     = azurerm_log_analytics_workspace.la.id
  log_analytics_destination_type = "Dedicated"

  enabled_log {
    category = "SQLSecurityAuditEvents"
  }

Note, this Terraform resource does have a storage_account_id property, but this doesn't seem to be necessary as storage is configured via the azurerm_mssql_server_extended_auditing_policy resource.

You would need separate azurerm_monitor_diagnostic_setting resources if you were configuring auditing per database though.

Common problems

The diagnostic setting can't find the master database

Error: creating Monitor Diagnostics Setting "diagnostic_setting" for Resource "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.Sql/servers/sql-terraform-sql-auditing-australiaeast/databases/master": unexpected status 404 (404 Not Found) with error: ResourceNotFound: The Resource 'Microsoft.Sql/servers/sql-terraform-sql-auditing-australiaeast/databases/master' under resource group 'rg-terraform-sql-auditing-australiaeast' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

It appears that sometimes the azurerm_mssql_server resource reports it is created, but the master database is not yet ready. The workaround is to add a dependency on another database resource - as by definition the master database must exist before any other user databases can be created.

Diagnostic setting fails to update with 409 Conflict

This error seems to happen to me when I try and set up Storage, Event Hubs and Log Analytics at the same time.

Error: creating Monitor Diagnostics Setting "diagnostic_setting" for Resource "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.Sql/servers/sql-terraform-sql-auditing-australiaeast/databases/master": unexpected status 409 (409 Conflict) with response: {"code":"Conflict","message":"Data sink '/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.EventHub/namespaces/evhns-terraform-sql-auditing-australiaeast/authorizationRules/evhar-terraform-sql-auditing-australiaeast' is already used in diagnostic setting 'SQLSecurityAuditEvents_3d229c42-c7e7-4c97-9a99-ec0d0d8b86c1' for category 'SQLSecurityAuditEvents'. Data sinks can't be reused in different settings on the same category for the same resource."}

After a lot of trial and error, I've found the solution is to add a depends_on block in your azurerm_mssql_server_extended_auditing_policy resource, so that the azurerm_monitor_diagnostic_setting is created first. (This feels like a bug in the Terraform AzureRM provider)

resource "azurerm_mssql_server_extended_auditing_policy" "auditing" {
  server_id = azurerm_mssql_server.mssql.id

  storage_endpoint  = azurerm_storage_account.storage.primary_blob_endpoint
  retention_in_days = 6

  depends_on = [azurerm_monitor_diagnostic_setting.mssql_server]
}

Switching from Storage access keys to managed identity has no effect

Removing the storage access key properties from azurerm_mssql_server_extended_auditing_policy doesn't currently switch the authentication to managed identity. The problem may relate to the storage_account_subscription_id property. This is an optional property and while you usually don't need to set it if the storage account is in the same subscription, it appears that the AzureRM provider is setting it on your behalf, such that when you remove the other access key properties it doesn't know to set this property to null.

If you know ahead of time that you'll be transitioning from access keys to managed identity, it might be worth setting storage_account_subscription_id first. Then later on, when you remove that and the other access_key properties maybe Terraform will do the right thing?

Solution resource

If you ever hit the Save button on the Azure SQL Auditing page, you may end up with a Solution resource being created for your auditing. This is useful, though it can cause problems if you are trying to destroy your Terraform resources, as it can put locks on the resources and Terraform doesn't know to destroy the solution resource first.

You could try to pre-emptively create the solution resource in Terraform. For example:

resource "azurerm_log_analytics_solution" "example" {
  solution_name         = "SQLAuditing"
  location              = data.azurerm_resource_group.rg.location
  resource_group_name   = data.azurerm_resource_group.rg.name
  workspace_resource_id = azurerm_log_analytics_workspace.la.id
  workspace_name        = azurerm_log_analytics_workspace.la.name

  plan {
    publisher = "Microsoft"
    product   = "SQLAuditing"
  }

  depends_on = [azurerm_monitor_diagnostic_setting.mssql_server]
}

Though it seems that when you use Terraform to create this resource, it names it SQLAuditing(log-terraform-sql-auditing-australiaeast), whereas if you use the portal, it is named SQLAuditing[log-terraform-sql-auditing-australiaeast].

So instead this looks like a good use for the AzApi provider and the azapi_resource

resource "azapi_resource" "symbolicname" {
  type      = "Microsoft.OperationsManagement/solutions@2015-11-01-preview"
  name      = "SQLAuditing[${azurerm_log_analytics_workspace.la.name}]"
  location  = data.azurerm_resource_group.rg.location
  parent_id = data.azurerm_resource_group.rg.id

  tags = {}
  body = {
    plan = {
      name          = "SQLAuditing[${azurerm_log_analytics_workspace.la.name}]"
      product       = "SQLAuditing"
      promotionCode = ""
      publisher     = "Microsoft"
    }
    properties = {
      containedResources = [
        "${azurerm_log_analytics_workspace.la.id}/views/SQLSecurityInsights",
        "${azurerm_log_analytics_workspace.la.id}/views/SQLAccessToSensitiveData"
      ]
      referencedResources = []
      workspaceResourceId = azurerm_log_analytics_workspace.la.id
    }
  }
}

Other troubleshooting tips

The Azure CLI can also be useful in checking what the current state of audit configuration is.

Here's two examples showing auditing configured for all three destinations:

az monitor diagnostic-settings list --resource /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.Sql/servers/sql-terraform-sql-auditing-australiaeast/databases/master

gives the following:

[
  {
    "eventHubAuthorizationRuleId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.EventHub/namespaces/evhns-terraform-sql-auditing-australiaeast/authorizationRules/evhar-terraform-sql-auditing-australiaeast",
    "eventHubName": "evh-terraform-sql-auditing-australiaeast",
    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-terraform-sql-auditing-australiaeast/providers/microsoft.sql/servers/sql-terraform-sql-auditing-australiaeast/databases/master/providers/microsoft.insights/diagnosticSettings/diagnostic_setting",
    "logs": [
      {
        "category": "SQLSecurityAuditEvents",
        "enabled": true,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "SQLInsights",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "AutomaticTuning",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "QueryStoreRuntimeStatistics",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "QueryStoreWaitStatistics",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "Errors",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "DatabaseWaitStatistics",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "Timeouts",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "Blocks",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "Deadlocks",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "DevOpsOperationsAudit",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      }
    ],
    "metrics": [
      {
        "category": "Basic",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "InstanceAndAppAdvanced",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      },
      {
        "category": "WorkloadManagement",
        "enabled": false,
        "retentionPolicy": {
          "days": 0,
          "enabled": false
        }
      }
    ],
    "name": "diagnostic_setting",
    "resourceGroup": "rg-terraform-sql-auditing-australiaeast",
    "type": "Microsoft.Insights/diagnosticSettings",
    "workspaceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.OperationalInsights/workspaces/log-terraform-sql-auditing-australiaeast"
  }
]

And the Azure SQL audit policy

az sql server audit-policy show -g rg-terraform-sql-auditing-australiaeast -n sql-terraform-sql-auditing-australiaeast

Gives

{
  "auditActionsAndGroups": [
    "SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP",
    "FAILED_DATABASE_AUTHENTICATION_GROUP",
    "BATCH_COMPLETED_GROUP"
  ],
  "blobStorageTargetState": "Enabled",
  "eventHubAuthorizationRuleId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.EventHub/namespaces/evhns-terraform-sql-auditing-australiaeast/authorizationRules/evhar-terraform-sql-auditing-australiaeast",
  "eventHubName": "evh-terraform-sql-auditing-australiaeast",
  "eventHubTargetState": "Enabled",
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.Sql/servers/sql-terraform-sql-auditing-australiaeast/auditingSettings/Default",
  "isAzureMonitorTargetEnabled": true,
  "isDevopsAuditEnabled": null,
  "isManagedIdentityInUse": true,
  "isStorageSecondaryKeyInUse": null,
  "logAnalyticsTargetState": "Enabled",
  "logAnalyticsWorkspaceResourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-terraform-sql-auditing-australiaeast/providers/Microsoft.OperationalInsights/workspaces/log-terraform-sql-auditing-australiaeast",
  "name": "Default",
  "queueDelayMs": null,
  "resourceGroup": "rg-terraform-sql-auditing-australiaeast",
  "retentionDays": 6,
  "state": "Enabled",
  "storageAccountAccessKey": null,
  "storageAccountSubscriptionId": "00000000-0000-0000-0000-000000000000",
  "storageEndpoint": "https://sttfsqlauditauew0o.blob.core.windows.net/",
  "type": "Microsoft.Sql/servers/auditingSettings"
}

Deploying to a new environment

The import statement will fail if you try to deploy to a brand new empty environment. It makes sense when you think about it - how can you import a resource that doesn't exist yet? Unfortunately, there's no way to make the import statement conditional.

An example of this would be if your application has been in development for a while, and to assist in testing you now want to create a new UAT environment. There are no resources in the UAT environment yet, so the import statement will fail.

Initializing plugins and modules...
data.azurerm_resource_group.group: Refreshing...
data.azurerm_client_config.client: Refreshing...
data.azurerm_client_config.client: Refresh complete after 0s [id=xxxxxxxxxxxxx=]
data.azurerm_resource_group.group: Refresh complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
╷
│ Error: Cannot import non-existent remote object
│
│ While attempting to import an existing object to
│ "azurerm_service_plan.plan", the provider detected that no object exists
│ with the given id. Only pre-existing objects can be imported; check that
│ the id is correct and that it is associated with the provider's configured
│ region or endpoint, or use "terraform apply" to create a new remote object
│ for this resource.
╵
Operation failed: failed running terraform plan (exit 1)

Deploying to an environment where resources may have been deleted

This is similar to the previous scenario. It's one I've encountered where we have non-production environments where you want to "mothball" when they're not being actively used to save money. We selectively delete resources, and then when the environment is needed we re-provision them. Again, the import statement will fail if it is referencing a deleted resource.

A Solution

The workaround to support these scenarios is to not use the import statement in your HCL code, but instead use the terraform import command. Because we're calling the command from the command line, we can make it conditional. e.g.

if az appservice plan show --name plan-tfupgrade-australiasoutheast --resource-group $ARM_RESOURCE_GROUP --query id --output tsv > /dev/null 2>&1; then
  terraform import azurerm_service_plan.plan /subscriptions/$ARM_SUBSCRIPTION_ID/resourceGroups/$ARM_RESOURCE_GROUP/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast
else
  echo "Resource plan-tfupgrade-australiasoutheast does not exist in Azure"
fi

Repeat this pattern for all resources that need to be imported. Also, take care when referencing the Azure resource names. They need to be correct for the detection code to work as expected!

It's not quite as elegant as using the import statement in your HCL code, but it does the job.

Example output

Here's an example of the output from the terraform import command:

data.azurerm_resource_group.group: Reading...
data.azurerm_client_config.client: Reading...
data.azurerm_client_config.client: Read complete after 0s [id=xxxxxxxxxxxx=]
data.azurerm_resource_group.group: Read complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Importing from ID "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"...
azurerm_service_plan.plan: Import prepared!
  Prepared azurerm_service_plan for import
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]

Import successful!

And the resultant plan shows no additional changes are necessary, which is just what we like to see!

data.azurerm_resource_group.group: Reading...
data.azurerm_client_config.client: Reading...
data.azurerm_client_config.client: Read complete after 0s [id=xxxxxxxxxxxxx=]
data.azurerm_resource_group.group: Read complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

And if we apply this to an empty environment, it also runs successfully!

azurerm_service_plan.plan: Creating...
azurerm_service_plan.plan: Still creating... [10s elapsed]
azurerm_service_plan.plan: Creation complete after 10s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Creating...
azurerm_linux_web_app.appservice: Still creating... [10s elapsed]
azurerm_linux_web_app.appservice: Still creating... [20s elapsed]
azurerm_linux_web_app.appservice: Still creating... [30s elapsed]
azurerm_linux_web_app.appservice: Still creating... [40s elapsed]
azurerm_linux_web_app.appservice: Still creating... [50s elapsed]
azurerm_linux_web_app.appservice: Creation complete after 56s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Post-migration clean up

Whether you use the import statement or the terraform import command, once you've migrated your Terraform resources in all existing environments, I recommend you remove the migration code from HCL and pipeline YAML files.

Remove the import statements as they have no further purpose, and you only risk encountering the issue mentioned above if you ever want to deploy to a new environment.

There's less risk of leaving the conditional terraform import commands in the pipeline YAML files, but as long as they remain they are slowing the pipeline down. Remove them to keep your pipelines running as fast as possible and to keep your YAML files as simple as possible. If you ever need to do another migration in the future, you'll have the commands in your source control history to refer to.

I've created an example repo that demonstrates the migration process outlined here. Find it at https://github.com/flcdrg/terraform-azure-upgrade-resources.

While this post uses Azure and Azure Pipelines, the same principles should apply for other cloud providers and CI/CD systems.

To set the scene, here's some Terraform code that creates an Azure App Service Plan and an App Service (src). The resource types are from v2.x AzureRM provider. Bear in mind, the last release of v2 was v2.99.0 in March 2022.

# https://registry.terraform.io/providers/hashicorp/azurerm/2.99.0/docs/resources/app_service_plan
resource "azurerm_app_service_plan" "plan" {
  name                = "plan-tfupgrade-australiasoutheast"
  resource_group_name = data.azurerm_resource_group.group.name
  location            = data.azurerm_resource_group.group.location
  kind                = "Linux"
  reserved            = true
  sku {
    tier = "Basic"
    size = "B1"
  }
}

# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service
resource "azurerm_app_service" "appservice" {
  app_service_plan_id = azurerm_app_service_plan.plan.id
  name                = "appservice-tfupgrade-australiasoutheast"
  location            = data.azurerm_resource_group.group.location
  resource_group_name = data.azurerm_resource_group.group.name
  https_only          = true

  app_settings = {
    "TEST" = "TEST"
  }

  site_config {
    always_on                 = true
    ftps_state                = "Disabled"
    http2_enabled             = true
    linux_fx_version          = "DOTNETCORE|6.0"
    min_tls_version           = "1.2"
    use_32_bit_worker_process = false
  }
  identity {
    type = "SystemAssigned"
  }
}

The documentation for AzureRM v3.x shows that these resource types are deprecated and will be completely removed in v4.x. In addition, as these resource types are not being updated, they don't support the latest features of Azure App Services, such as the new .NET 8 runtime.

So how can we switch to the azurerm_service_plan and azurerm_linux_web_app resource types?

You might think it's just a matter of just changing the resource types and updating a few properties. But if you tried that you'll discover that Terraform will try to delete the existing resources and create new ones. This is because the resource types are different, and Terraform doesn't know that they are actually the same thing because the state representation of those resources is different.

Instead, we need to let Terraform know that the Azure resources that have already been created map to the new Terraform resource types we've defined in our configuration. In addition, we want to do this in a testable way using a pull request to verify that our changes look correct before we merge them into the main branch.

The approach we'll take is to make use of the relatively new import block language feature. (In a future blog post I'll cover when you might consider using the terraform import CLI command instead).

By using the import block, we can tell Terraform that the existing resources in Azure should be mapped to the new resource types we've defined in Terraform configuration. This means that Terraform will not try to delete the existing resources and create new ones. Instead, it will update the existing resources to match the Terraform configuration.

In the following example, we're indicating that the Azure resource with the resource ID /subscriptions/.../resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast should be mapped to the azurerm_service_plan resource type. Note the use of the data block reference to insert the subscription ID, rather than hard-coding it.

import {
  id = "/subscriptions/${data.azurerm_client_config.client.subscription_id}/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
  to = azurerm_service_plan.plan
}

resource "azurerm_service_plan" "plan" {
  name                = "plan-tfupgrade-australiasoutheast"
  resource_group_name = data.azurerm_resource_group.group.name
  location            = data.azurerm_resource_group.group.location
  sku_name            = "B1"
  os_type             = "Linux"
}

Often when you're adding the new resource, the property names and 'shape' will change. Sometimes it's pretty easy to figure out the equivalent, but sometimes you might need some help. One option you can utilise is generating the configuration.

In this case, you add the import block, but don't add the resource block. If you then run terraform plan -generate-config-out=generated_resources.tf. Terraform will create a new file generated_resources.tf which will contain generated resources. You can then copy/paste those over into your regular .tf files. You'll almost certainly want to edit them to remove redundant settings and replace hard-coded values with variable references where applicable. If you're doing this as part of a pipeline, publish the generated file as a build artifact, so you can download it and incorporate the changes. You could make this an optional part of the pipeline that is enabled by setting a pipeline parameter to true.

There's still one problem to solve though. While we've mapped the new resource types to the existing resources, Terraform state still knows about the old resource types, and will try to delete them now that they are no longer defined in the Terraform configuration. To solve this, we can use the terraform state rm command to remove the old resources from state. If they're not in state, then Terraform doesn't know about them and won't try to delete them.

The following script will remove the old resources from state if they exist. Note that the terraform state rm command will fail if the resource doesn't exist in state, so we need to check for the existence of the resource first.

# Remove state of old resources from Terraform
mapfile -t RESOURCES < <( terraform state list )

if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service_plan.plan" ]]; then
  terraform state rm azurerm_app_service_plan.plan
fi

if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service.appservice" ]]; then
  terraform state rm azurerm_app_service.appservice
fi

You will need to add an entry in this script for each Terraform resource type that you are removing.

Testing

Ok, so we have a strategy for upgrading our Terraform resources. But how do we test it? We don't want to just merge the changes into the main branch and hope for the best. We want to test it first, and do this in isolation from other changes that might be happening in the main branch (or other branches). To test our changes need to update the Terraform state. But if we update the state used by everyone else then we won't be popular when their builds start failing because Terraform will be trying to recreate resources that we've just deleted from state. Except those resources still exist in Azure!

What we want is a local copy of the Terraform state that we can try out our changes in without affecting anyone else. One way to do this is to copy the remote state to a local file, then reinitialise Terraform to use the 'local' backend. Obviously we won't do a real deployment using this, but it is perfect for running terraform plan against.

Here's an example script that will copy the remote state to a local file, then reinitialise Terraform to use the local backend. It assumes that your backend configuration is defined separately in a backend.tf file. Normally this would be pointing to a remote backend (e.g. Terraform Cloud or an Azure Storage account), within the pipeline run we replace this file with configuration to use a local backend.

terraform state pull > $(Build.ArtifactStagingDirectory)/pull.tfstate

cat > backend.tf <<EOF
terraform {
  backend "local" {
    path = "$(Build.ArtifactStagingDirectory)/pull.tfstate"
  }
}
EOF

# Reset Terraform to use local backend
terraform init -reconfigure -no-color -input=false

We now should have all the pieces in place to test our changes on PR build, and then once we're happy with the plan, merge the changes and run the migration for real. If you have multiple environments (dev/test/prod) then you can roll this out to each environment as part of the normal release process.

For Azure Pipelines, we make use of conditional expressions, so that on PR builds we test the migration using local state, but on the main branch we modify the remote state and actually apply the changes.

Here's the Azure Pipeline in full (src):

trigger: none

pr:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

variables:
  - group: Terraform-Token

jobs:
  - job: build
    displayName: "Test Terraform Upgrade"

    variables:
      TerraformSourceDirectory: $(System.DefaultWorkingDirectory)/v3

    steps:
      - script: echo "##vso[task.setvariable variable=TF_TOKEN_app_terraform_io]$(TF_TOKEN)"
        displayName: "Terraform Token"

      - task: TerraformInstaller@2
        displayName: "Terraform: Installer"
        inputs:
          terraformVersion: "latest"

      - task: TerraformCLI@2
        displayName: "Terraform: init"
        inputs:
          command: init
          workingDirectory: "$(TerraformSourceDirectory)"
          backendType: selfConfigured
          commandOptions: -no-color -input=false
          allowTelemetryCollection: false

      - ${{ if ne(variables['Build.SourceBranch'], 'refs/heads/main') }}:
          # Copy state from Terraform Cloud to local, so we can modify it without affecting the remote state
          - script: |
              terraform state pull > $(Build.ArtifactStagingDirectory)/pull.tfstate

              # Write multiple lines of text to local file using bash
              cat > backend.tf <<EOF
              terraform {
                backend "local" {
                  path = "$(Build.ArtifactStagingDirectory)/pull.tfstate"
                }
              }
              EOF

              # Reset Terraform to use local backend
              terraform init -reconfigure -no-color -input=false
            displayName: "Script: Use Terraform Local Backend"
            workingDirectory: $(TerraformSourceDirectory)

      - script: |
          # Remove state of old resources from Terraform
          mapfile -t RESOURCES < <( terraform state list )

          if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service_plan.plan" ]]; then
            terraform state rm azurerm_app_service_plan.plan
          fi

          if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service.appservice" ]]; then
            terraform state rm azurerm_app_service.appservice
          fi
        displayName: "Script: Remove old resources from Terraform State"
        workingDirectory: $(TerraformSourceDirectory)

      - task: TerraformCLI@2
        displayName: "Terraform: validate"
        inputs:
          command: validate
          workingDirectory: "$(TerraformSourceDirectory)"
          commandOptions: -no-color

      - ${{ if ne(variables['Build.SourceBranch'], 'refs/heads/main') }}:
          - task: TerraformCLI@2
            displayName: "Terraform: plan"
            inputs:
              command: plan
              workingDirectory: "$(TerraformSourceDirectory)"
              commandOptions: -no-color -input=false -detailed-exitcode
              environmentServiceName: Azure MSDN - rg-tfupgrade-australiasoutheast
              publishPlanResults: Plan
              allowTelemetryCollection: false

      - ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
          - task: TerraformCLI@2
            displayName: "Terraform: apply"
            inputs:
              command: apply
              workingDirectory: "$(TerraformSourceDirectory)"
              commandOptions: -no-color -input=false -auto-approve
              allowTelemetryCollection: false

Using it in practise

Ideally when you migrate the resource types, there will be no changes to the properties (and Terraform will report that no changes need to be made). Often the new resource type provides additional properties that you can take advantage of. Whether you set this initially or in a subsequent PR is up to you.

Here's an example output from terraform plan:

Terraform v1.6.4
on linux_amd64
Initializing plugins and modules...
data.azurerm_resource_group.group: Refreshing...
data.azurerm_client_config.client: Refreshing...
data.azurerm_client_config.client: Refresh complete after 0s [id=xxxxxxxxxxx=]
data.azurerm_resource_group.group: Refresh complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

Terraform will perform the following actions:

  # azurerm_linux_web_app.appservice will be imported
    resource "azurerm_linux_web_app" "appservice" {
        app_settings                                   = {
            "TEST" = "TEST"
        }
        client_affinity_enabled                        = false
        client_certificate_enabled                     = false
        client_certificate_mode                        = "Required"
        custom_domain_verification_id                  = (sensitive value)
        default_hostname                               = "appservice-tfupgrade-australiasoutheast.azurewebsites.net"
        enabled                                        = true
        ftp_publish_basic_authentication_enabled       = true
        https_only                                     = true
        id                                             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast"
        key_vault_reference_identity_id                = "SystemAssigned"
        kind                                           = "app,linux"
        location                                       = "australiasoutheast"
        name                                           = "appservice-tfupgrade-australiasoutheast"
        outbound_ip_address_list                       = [
            "52.189.223.107",
            "13.77.42.25",
            "13.77.46.217",
            "52.189.221.141",
            "13.77.50.99",
        ]
        outbound_ip_addresses                          = "52.189.223.107,13.77.42.25,13.77.46.217,52.189.221.141,13.77.50.99"
        possible_outbound_ip_address_list              = [
            "52.189.223.107",
            "13.77.42.25",
            "13.77.46.217",
            "52.189.221.141",
            "52.243.85.201",
            "52.243.85.94",
            "52.189.234.152",
            "13.77.56.61",
            "52.189.214.112",
            "20.11.210.198",
            "20.211.233.197",
            "20.211.238.191",
            "20.11.210.187",
            "20.11.211.1",
            "20.11.211.80",
            "4.198.70.38",
            "20.92.41.250",
            "4.198.68.27",
            "4.198.68.42",
            "20.92.47.59",
            "20.92.42.78",
            "13.77.50.99",
        ]
        possible_outbound_ip_addresses                 = "52.189.223.107,13.77.42.25,13.77.46.217,52.189.221.141,52.243.85.201,52.243.85.94,52.189.234.152,13.77.56.61,52.189.214.112,20.11.210.198,20.211.233.197,20.211.238.191,20.11.210.187,20.11.211.1,20.11.211.80,4.198.70.38,20.92.41.250,4.198.68.27,4.198.68.42,20.92.47.59,20.92.42.78,13.77.50.99"
        public_network_access_enabled                  = true
        resource_group_name                            = "rg-tfupgrade-australiasoutheast"
        service_plan_id                                = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
        site_credential                                = (sensitive value)
        tags                                           = {}
        webdeploy_publish_basic_authentication_enabled = true

        identity {
            identity_ids = []
            principal_id = "a71e1fd5-e61b-4591-a439-98bad90cc837"
            tenant_id    = "59b0934d-4f35-4bff-a2b7-a451fe5f8bd6"
            type         = "SystemAssigned"
        }

        site_config {
            always_on                               = true
            auto_heal_enabled                       = false
            container_registry_use_managed_identity = false
            default_documents                       = []
            detailed_error_logging_enabled          = false
            ftps_state                              = "Disabled"
            health_check_eviction_time_in_min       = 0
            http2_enabled                           = true
            linux_fx_version                        = "DOTNETCORE|6.0"
            load_balancing_mode                     = "LeastRequests"
            local_mysql_enabled                     = false
            managed_pipeline_mode                   = "Integrated"
            minimum_tls_version                     = "1.2"
            remote_debugging_enabled                = false
            remote_debugging_version                = "VS2019"
            scm_minimum_tls_version                 = "1.2"
            scm_type                                = "VSTSRM"
            scm_use_main_ip_restriction             = false
            use_32_bit_worker                       = false
            vnet_route_all_enabled                  = false
            websockets_enabled                      = false
            worker_count                            = 1

            application_stack {
                docker_registry_password = (sensitive value)
                dotnet_version           = "6.0"
            }
        }
    }

  # azurerm_service_plan.plan will be imported
    resource "azurerm_service_plan" "plan" {
        id                           = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
        kind                         = "linux"
        location                     = "australiasoutheast"
        maximum_elastic_worker_count = 1
        name                         = "plan-tfupgrade-australiasoutheast"
        os_type                      = "Linux"
        per_site_scaling_enabled     = false
        reserved                     = true
        resource_group_name          = "rg-tfupgrade-australiasoutheast"
        sku_name                     = "B1"
        tags                         = {}
        worker_count                 = 1
        zone_balancing_enabled       = false
    }

Plan: 2 to import, 0 to add, 0 to change, 0 to destroy.

In the next post, I'll cover a few things to watch out for, and some post-migration clean up steps.

- task: TerraformCLI@2
  inputs:
    command: init
    workingDirectory: "$(System.DefaultWorkingDirectory)/v2"
    backendType: selfConfigured
    commandOptions: -no-color -input=false
    allowTelemetryCollection: false

and it kept failing with the error:

/opt/hostedtoolcache/terraform/1.6.4/x64/terraform version
Terraform v1.6.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v2.99.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
/opt/hostedtoolcache/terraform/1.6.4/x64/terraform init --input=false --no-color
Usage: terraform [global options] init [options]
...
Terraform command 'init' failed with exit code '1'.

I tried all sorts of things. It looked identical to other pipelines I had working (and ones I'd seen online). Was there a weird invisible character being passed on the command line? Out of desperation I copied the commandOptions line from another pipeline (which looked identical except that the arguments were in a different order).

But when I looked at the diff, not only were the arguments different, I realised that the dashes were different too! Terraform CLI arguments use a single dash, not a double dash. So the correct line is

commandOptions: -no-color -input=false

In hindsight, I realised that this is the first pipeline I've written on GitHub using the Terraform CLI (e.g. not in a work context) and so I did it manually, rather than copy/pasting from an existing (working) pipeline. A pity Terraform doesn't support double dashes, but there you go.