Terraform Modules and CI/CD for Data Engineers: Writing Reusable Modules, Module Composition, Registry and Versioning, Deploying Through Azure DevOps YAML Pipelines, Plan-Apply Pattern, Pipeline Templates, and Terraform vs DABs

Leave a Comment / Azure, Data Engineering, DevOps

Table of Contents

In the previous post, we covered Terraform fundamentals — HCL, providers, resources, state, and provisioning a data platform. That single-file approach works for small projects, but as your infrastructure grows (multiple storage accounts, several Databricks workspaces, networking, monitoring), a flat main.tf becomes unmanageable. Modules solve this by packaging related resources into reusable, testable components.

This post covers writing production-grade modules and deploying them through the YAML pipelines we built in the YAML Pipelines Deep Dive — creating a complete infrastructure CI/CD pipeline.

Analogy — LEGO sets vs loose bricks. Writing all resources in one main.tf is like building with loose LEGO bricks — you can build anything, but it is slow, messy, and impossible to rebuild consistently. A Terraform module is a LEGO set — a pre-packaged collection of bricks (resources) with instructions (variables and outputs) that builds one specific thing (a storage account, a Databricks workspace). You build a castle by combining sets: the wall set, the tower set, the gate set. Each set is tested independently, versioned, and reusable across projects.

Why Modules — The Problem with Flat Terraform

Without modules (flat structure):

  main.tf (500+ lines)
    - Resource group
    - Storage account 1 (data lake)
    - Storage account 2 (terraform state)
    - 12 storage containers
    - Key Vault
    - 5 Key Vault access policies
    - 8 Key Vault secrets
    - Databricks workspace
    - Databricks access connector
    - 3 RBAC role assignments
    - Databricks cluster
    - Databricks SQL warehouse
    - Unity Catalog storage credential
    - Unity Catalog external location
    - Unity Catalog catalog
    - 3 Unity Catalog schemas
    - ...and growing

  Problems:
    - Hard to read (500+ lines in one file)
    - Hard to test (can't test storage independently from Databricks)
    - Hard to reuse (copy-paste between projects)
    - Hard to review (PRs touch everything)
    - Hard to divide work (two engineers can't work on different parts)

With modules:

  main.tf (50 lines -- just module calls)
    module "storage"    { source = "./modules/storage" }
    module "keyvault"   { source = "./modules/keyvault" }
    module "databricks" { source = "./modules/databricks" }
    module "unity_catalog" { source = "./modules/unity_catalog" }

  Each module: 50-100 lines, self-contained, testable, reusable

Module Anatomy — Structure, Inputs, Outputs

Every module follows the same structure:

modules/storage/
  ├── main.tf          # Resources (storage account, containers)
  ├── variables.tf     # Inputs (what the caller provides)
  ├── outputs.tf       # Outputs (what the module exposes)
  └── README.md        # Documentation (optional but recommended)

Rules:
  - A module is just a directory with .tf files
  - Modules have their own variables and outputs (separate from root)
  - Modules do NOT have provider blocks (inherited from root)
  - Modules do NOT have backend blocks (only root has the backend)
  - Modules should do ONE thing well (single responsibility)

Building a Storage Account Module

# modules/storage/variables.tf
variable "name" {
  description = "Storage account name (must be globally unique)"
  type        = string
}

variable "resource_group_name" {
  description = "Resource group to create storage account in"
  type        = string
}

variable "location" {
  description = "Azure region"
  type        = string
}

variable "environment" {
  description = "Environment (dev, staging, prod)"
  type        = string
}

variable "containers" {
  description = "List of storage containers to create"
  type        = list(string)
  default     = ["bronze", "silver", "gold"]
}

variable "replication_type" {
  description = "Storage replication type"
  type        = string
  default     = "LRS"
}

variable "tags" {
  description = "Tags to apply to all resources"
  type        = map(string)
  default     = {}
}
# modules/storage/main.tf
resource "azurerm_storage_account" "this" {
  name                     = var.name
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = var.replication_type
  account_kind             = "StorageV2"
  is_hns_enabled           = true    # ADLS Gen2

  tags = var.tags
}

resource "azurerm_storage_container" "this" {
  for_each              = toset(var.containers)
  name                  = each.value
  storage_account_id    = azurerm_storage_account.this.id
  container_access_type = "private"
}
# modules/storage/outputs.tf
output "id" {
  description = "Storage account ID"
  value       = azurerm_storage_account.this.id
}

output "name" {
  description = "Storage account name"
  value       = azurerm_storage_account.this.name
}

output "primary_access_key" {
  description = "Primary access key"
  value       = azurerm_storage_account.this.primary_access_key
  sensitive   = true
}

output "primary_dfs_endpoint" {
  description = "Primary DFS endpoint (for ADLS Gen2)"
  value       = azurerm_storage_account.this.primary_dfs_endpoint
}

Building a Databricks Workspace Module

# modules/databricks/variables.tf
variable "name" {
  description = "Databricks workspace name"
  type        = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

variable "sku" {
  description = "Databricks SKU (standard, premium, trial)"
  type        = string
  default     = "premium"
}

variable "tags" {
  type    = map(string)
  default = {}
}
# modules/databricks/main.tf
resource "azurerm_databricks_access_connector" "this" {
  name                = "ac-${var.name}"
  resource_group_name = var.resource_group_name
  location            = var.location

  identity {
    type = "SystemAssigned"
  }

  tags = var.tags
}

resource "azurerm_databricks_workspace" "this" {
  name                = var.name
  resource_group_name = var.resource_group_name
  location            = var.location
  sku                 = var.sku

  tags = var.tags
}
# modules/databricks/outputs.tf
output "workspace_url" {
  value = azurerm_databricks_workspace.this.workspace_url
}

output "workspace_id" {
  value = azurerm_databricks_workspace.this.id
}

output "workspace_resource_id" {
  value = azurerm_databricks_workspace.this.id
}

output "access_connector_id" {
  value = azurerm_databricks_access_connector.this.id
}

output "access_connector_principal_id" {
  value = azurerm_databricks_access_connector.this.identity[0].principal_id
}

Building a Key Vault Module

# modules/keyvault/variables.tf
variable "name" {
  type = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

variable "tenant_id" {
  type = string
}

variable "admin_object_ids" {
  description = "Object IDs that get full secret access"
  type        = list(string)
  default     = []
}

variable "secrets" {
  description = "Map of secret names to values"
  type        = map(string)
  default     = {}
  sensitive   = true
}

variable "purge_protection" {
  type    = bool
  default = false
}

variable "tags" {
  type    = map(string)
  default = {}
}
# modules/keyvault/main.tf
resource "azurerm_key_vault" "this" {
  name                     = var.name
  resource_group_name      = var.resource_group_name
  location                 = var.location
  tenant_id                = var.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = var.purge_protection

  tags = var.tags
}

resource "azurerm_key_vault_access_policy" "admins" {
  for_each = toset(var.admin_object_ids)

  key_vault_id = azurerm_key_vault.this.id
  tenant_id    = var.tenant_id
  object_id    = each.value

  secret_permissions = ["Get", "List", "Set", "Delete", "Purge"]
}

resource "azurerm_key_vault_secret" "this" {
  for_each = var.secrets

  name         = each.key
  value        = each.value
  key_vault_id = azurerm_key_vault.this.id

  depends_on = [azurerm_key_vault_access_policy.admins]
}
# modules/keyvault/outputs.tf
output "id" {
  value = azurerm_key_vault.this.id
}

output "vault_uri" {
  value = azurerm_key_vault.this.vault_uri
}

output "name" {
  value = azurerm_key_vault.this.name
}

Calling Modules from the Root Configuration

# root main.tf -- clean, readable, 60 lines
data "azurerm_client_config" "current" {}

# Resource Group
resource "azurerm_resource_group" "main" {
  name     = "rg-${local.name_prefix}"
  location = var.location
  tags     = local.common_tags
}

# Data Lake Storage
module "datalake" {
  source = "./modules/storage"

  name                = "st${replace(local.name_prefix, "-", "")}lake"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  environment         = var.environment
  containers          = ["bronze", "silver", "gold", "landing"]
  replication_type    = var.environment == "prod" ? "GRS" : "LRS"
  tags                = local.common_tags
}

# Key Vault
module "keyvault" {
  source = "./modules/keyvault"

  name                = "kv-${local.name_prefix}"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  tenant_id           = data.azurerm_client_config.current.tenant_id
  purge_protection    = var.environment == "prod"
  admin_object_ids    = [data.azurerm_client_config.current.object_id]
  tags                = local.common_tags

  secrets = {
    "storage-account-key"  = module.datalake.primary_access_key
    "storage-account-name" = module.datalake.name
  }
}

# Databricks Workspace
module "databricks" {
  source = "./modules/databricks"

  name                = "dbw-${local.name_prefix}"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  sku                 = var.databricks_sku
  tags                = local.common_tags
}

# RBAC: Grant Access Connector access to storage
resource "azurerm_role_assignment" "connector_storage" {
  scope                = module.datalake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = module.databricks.access_connector_principal_id
}
How modules communicate:

  module.datalake.primary_access_key  → passed to module.keyvault as a secret
  module.datalake.id                  → passed to RBAC role assignment
  module.databricks.access_connector_principal_id → passed to RBAC

  Data flows through outputs and inputs:
    Module A creates a resource → exposes attribute via output
    Root configuration reads the output → passes it as input to Module B
    Module B uses the input → creates dependent resources

  Terraform automatically builds the dependency graph:
    Resource Group → Storage Account → Key Vault (needs storage key)
                  → Databricks Workspace → RBAC (needs both)

Module Composition — Building a Complete Data Platform

# For larger platforms, compose modules into a "platform" module

# modules/data_platform/main.tf
module "storage" {
  source = "../storage"
  name   = "${var.name_prefix}-lake"
  # ... pass through variables
}

module "keyvault" {
  source = "../keyvault"
  name   = "${var.name_prefix}-kv"
  secrets = {
    "storage-key" = module.storage.primary_access_key
  }
  # ... pass through variables
}

module "databricks" {
  source = "../databricks"
  name   = "${var.name_prefix}-dbw"
  # ... pass through variables
}

# Root main.tf becomes ONE line:
module "data_platform" {
  source      = "./modules/data_platform"
  name_prefix = "dp-${var.environment}"
  location    = var.location
  environment = var.environment
  tags        = local.common_tags
}

Module Registry and Versioning

Three ways to source modules:

  1. Local path (development):
     source = "./modules/storage"
     Fastest for development, no versioning

  2. Git repository (team sharing):
     source = "git::https://dev.azure.com/MyCompany/Infra/_git/tf-modules//modules/storage?ref=v1.2.0"
     Version-controlled, shareable across projects
     Tag releases (v1.0.0, v1.1.0, v2.0.0)

  3. Terraform Registry (public or private):
     source  = "registry.terraform.io/myorg/storage/azurerm"
     version = "~> 1.2"
     Best for mature, widely-used modules

  Versioning best practice:
    - Use semantic versioning (major.minor.patch)
    - Pin to minor version: version = "~> 1.2" (allows 1.2.x)
    - Breaking changes = new major version
    - Test module changes before updating consumers

Deploying Terraform Through Azure DevOps Pipelines

This is where everything connects: Terraform modules deployed through YAML pipelines with approval gates.

# pipelines/terraform-ci.yml
# CI: Runs on every PR -- validates and plans
trigger: none
pr:
  branches:
    include:
      - main
  paths:
    include:
      - terraform/**

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: 'terraform-credentials'

steps:
  - script: |
      curl -fsSL https://releases.hashicorp.com/terraform/1.9.0/terraform_1.9.0_linux_amd64.zip -o tf.zip
      unzip -o tf.zip && sudo mv terraform /usr/local/bin/
    displayName: 'Install Terraform'

  - script: |
      cd terraform/
      terraform fmt -check -recursive
    displayName: 'Check formatting'

  - script: |
      cd terraform/
      terraform init -backend=false
      terraform validate
    displayName: 'Validate configuration'

  - script: |
      cd terraform/
      terraform init -backend-config="environments/dev/backend.conf"
      terraform plan -var-file="environments/dev/dev.tfvars" -input=false
    displayName: 'Terraform Plan (dev)'
    env:
      ARM_CLIENT_ID: $(ARM_CLIENT_ID)
      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
      ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
      ARM_TENANT_ID: $(ARM_TENANT_ID)

The Complete CI/CD Pipeline for Terraform

# pipelines/terraform-cd.yml
# CD: Runs on merge to main -- plans, approves, applies
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - terraform/**

variables:
  - name: terraformVersion
    value: '1.9.0'

stages:
  # Stage 1: Plan for Dev
  - stage: PlanDev
    displayName: 'Plan Dev'
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'terraform-dev-credentials'
    jobs:
      - job: Plan
        steps:
          - script: |
              curl -fsSL https://releases.hashicorp.com/terraform/$(terraformVersion)/terraform_$(terraformVersion)_linux_amd64.zip -o tf.zip
              unzip -o tf.zip && sudo mv terraform /usr/local/bin/
            displayName: 'Install Terraform'
          - script: |
              cd terraform/
              terraform init -backend-config="environments/dev/backend.conf"
              terraform plan -var-file="environments/dev/dev.tfvars" -out=dev.tfplan -input=false
            displayName: 'Terraform Plan'
            env:
              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
              ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
              ARM_TENANT_ID: $(ARM_TENANT_ID)
          - publish: terraform/dev.tfplan
            artifact: dev-plan

  # Stage 2: Apply to Dev (auto-approve)
  - stage: ApplyDev
    displayName: 'Apply Dev'
    dependsOn: PlanDev
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'terraform-dev-credentials'
    jobs:
      - deployment: Apply
        environment: 'dev'
        strategy:
          runOnce:
            deploy:
              steps:
                - download: current
                  artifact: dev-plan
                - script: |
                    cd terraform/
                    terraform init -backend-config="environments/dev/backend.conf"
                    terraform apply -input=false $(Pipeline.Workspace)/dev-plan/dev.tfplan
                  displayName: 'Terraform Apply'
                  env:
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

  # Stage 3: Plan for Prod
  - stage: PlanProd
    displayName: 'Plan Prod'
    dependsOn: ApplyDev
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'terraform-prod-credentials'
    jobs:
      - job: Plan
        steps:
          - script: |
              curl -fsSL https://releases.hashicorp.com/terraform/$(terraformVersion)/terraform_$(terraformVersion)_linux_amd64.zip -o tf.zip
              unzip -o tf.zip && sudo mv terraform /usr/local/bin/
            displayName: 'Install Terraform'
          - script: |
              cd terraform/
              terraform init -backend-config="environments/prod/backend.conf"
              terraform plan -var-file="environments/prod/prod.tfvars" -out=prod.tfplan -input=false
            displayName: 'Terraform Plan'
            env:
              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
              ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
              ARM_TENANT_ID: $(ARM_TENANT_ID)
          - publish: terraform/prod.tfplan
            artifact: prod-plan

  # Stage 4: Apply to Prod (requires approval)
  - stage: ApplyProd
    displayName: 'Apply Prod'
    dependsOn: PlanProd
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'terraform-prod-credentials'
    jobs:
      - deployment: Apply
        environment: 'production'    # Requires 2 approvers
        strategy:
          runOnce:
            deploy:
              steps:
                - download: current
                  artifact: prod-plan
                - script: |
                    cd terraform/
                    terraform init -backend-config="environments/prod/backend.conf"
                    terraform apply -input=false $(Pipeline.Workspace)/prod-plan/prod.tfplan
                  displayName: 'Terraform Apply'
                  env:
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

Pipeline Templates for Terraform

# pipelines/templates/terraform-plan.yml
parameters:
  - name: environment
    type: string
  - name: credentialsGroup
    type: string
  - name: terraformVersion
    type: string
    default: '1.9.0'

jobs:
  - job: Plan_${{ parameters.environment }}
    displayName: 'Plan ${{ parameters.environment }}'
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: ${{ parameters.credentialsGroup }}
    steps:
      - script: |
          curl -fsSL https://releases.hashicorp.com/terraform/${{ parameters.terraformVersion }}/terraform_${{ parameters.terraformVersion }}_linux_amd64.zip -o tf.zip
          unzip -o tf.zip && sudo mv terraform /usr/local/bin/
        displayName: 'Install Terraform'
      - script: |
          cd terraform/
          terraform init -backend-config="environments/${{ parameters.environment }}/backend.conf"
          terraform plan -var-file="environments/${{ parameters.environment }}/${{ parameters.environment }}.tfvars" -out=${{ parameters.environment }}.tfplan -input=false
        displayName: 'Terraform Plan'
        env:
          ARM_CLIENT_ID: $(ARM_CLIENT_ID)
          ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
          ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
          ARM_TENANT_ID: $(ARM_TENANT_ID)
      - publish: terraform/${{ parameters.environment }}.tfplan
        artifact: ${{ parameters.environment }}-plan
# Main pipeline using templates -- DRY and clean
trigger:
  branches:
    include: [main]
  paths:
    include: [terraform/**]

stages:
  - stage: PlanDev
    jobs:
      - template: templates/terraform-plan.yml
        parameters:
          environment: 'dev'
          credentialsGroup: 'terraform-dev-credentials'

  - stage: ApplyDev
    dependsOn: PlanDev
    jobs:
      - template: templates/terraform-apply.yml
        parameters:
          environment: 'dev'
          credentialsGroup: 'terraform-dev-credentials'

  - stage: PlanProd
    dependsOn: ApplyDev
    jobs:
      - template: templates/terraform-plan.yml
        parameters:
          environment: 'prod'
          credentialsGroup: 'terraform-prod-credentials'

  - stage: ApplyProd
    dependsOn: PlanProd
    jobs:
      - template: templates/terraform-apply.yml
        parameters:
          environment: 'prod'
          credentialsGroup: 'terraform-prod-credentials'

Multi-Environment CI/CD with Each Loops

# Dynamic multi-environment pipeline using each loop
parameters:
  - name: environments
    type: object
    default:
      - name: dev
        credentials: terraform-dev-credentials
        approvals: false
      - name: staging
        credentials: terraform-staging-credentials
        approvals: true
      - name: prod
        credentials: terraform-prod-credentials
        approvals: true

trigger:
  branches:
    include: [main]
  paths:
    include: [terraform/**]

stages:
  - ${{ each env in parameters.environments }}:
    - stage: Plan_${{ env.name }}
      displayName: 'Plan ${{ env.name }}'
      ${{ if ne(env.name, 'dev') }}:
        dependsOn: Apply_${{ variables.previousEnv }}
      jobs:
        - template: templates/terraform-plan.yml
          parameters:
            environment: ${{ env.name }}
            credentialsGroup: ${{ env.credentials }}

    - stage: Apply_${{ env.name }}
      displayName: 'Apply ${{ env.name }}'
      dependsOn: Plan_${{ env.name }}
      jobs:
        - template: templates/terraform-apply.yml
          parameters:
            environment: ${{ env.name }}
            credentialsGroup: ${{ env.credentials }}

Terraform vs Databricks Asset Bundles — When to Use Which

AspectTerraformDatabricks Asset Bundles (DABs)
What it managesAzure infrastructure (workspaces, storage, networking, IAM)Databricks resources (notebooks, jobs, pipelines, clusters)
LanguageHCLYAML (databricks.yml)
ScopeMulti-cloud (Azure, AWS, GCP)Databricks-only
State managementRemote state fileNo state file (API-driven)
Deploymentterraform plan/applydatabricks bundle deploy
Best forInfrastructure: resource groups, storage accounts, workspaces, Key Vaults, networkingWorkloads: jobs, notebooks, Lakeflow pipelines, ML models
Used together?YES — Terraform creates the workspace, DABs deploys workloads into it

The recommended separation:

  Terraform manages:
    - Resource groups
    - Storage accounts (ADLS Gen2)
    - Databricks workspaces
    - Access connectors and RBAC
    - Key Vaults
    - Unity Catalog metastore-level objects (credentials, external locations)
    - Networking (VNets, NSGs, private endpoints)

  DABs manages:
    - Databricks jobs and workflows
    - Notebooks
    - Lakeflow pipelines
    - Cluster configurations
    - Catalog, schema, and table definitions
    - ML model deployment

  CI/CD flow:
    1. Terraform pipeline creates/updates infrastructure (runs less frequently)
    2. DABs pipeline deploys workloads into the infrastructure (runs more frequently)
    3. Both triggered by changes in their respective directories

Common Mistakes

  1. Writing monolithic modules that do too much. A module that creates a storage account, Key Vault, Databricks workspace, and networking is not reusable — it is just your main.tf in a different directory. Each module should create one logical component. Compose modules in the root configuration.

  2. Not defining outputs for inter-module communication. If Module A creates a storage account but does not output the ID, Module B cannot reference it. Design module outputs as the API contract — every attribute another module might need should be an output.

  3. Hardcoding provider versions in modules. Modules should declare required providers but NOT pin versions. The root configuration controls versions. A module with version = "4.0.0" forces all consumers to use that exact version, creating conflicts.

  4. Applying Terraform without saving the plan. Running terraform apply without -out=plan.tfplan means the plan is regenerated at apply time. If resources changed between your review and apply (someone else modified Azure), the actual changes differ from what you reviewed. Always plan to a file and apply that file.

  5. Not creating separate service principals per environment. Using one service principal for dev and prod means a misconfigured pipeline can modify production infrastructure. Create dedicated service principals: sp-terraform-dev, sp-terraform-prod, each with access only to their resource group.

  6. Skipping terraform fmt and validate in CI. Format checking catches inconsistent indentation and style. Validate catches syntax errors and missing references. Both should run on every PR before planning. They are fast (seconds) and catch issues early.

  7. Using Terraform for Databricks workloads (jobs, notebooks) instead of DABs. Terraform manages infrastructure well but is awkward for deploying notebooks and job configurations that change frequently. Use DABs for Databricks workloads — it is purpose-built for that use case, requires no state file, and integrates natively with the Databricks CLI.

  8. Not protecting production state with lifecycle rules. Without lifecycle { prevent_destroy = true } on critical resources (production storage accounts, databases), a mistyped terraform destroy or removed resource block destroys production data. Add prevent_destroy to every production resource that holds data.

Interview Questions

Q: What are Terraform modules and why are they important? A: Modules are reusable packages of Terraform configuration that encapsulate related resources with defined inputs (variables) and outputs. They solve code organization (break 500-line files into focused components), reusability (same module for dev and prod), testability (test a storage module independently), and team collaboration (different engineers work on different modules). A module is simply a directory with .tf files, called from the root configuration with a module block.

Q: How do modules communicate with each other? A: Modules communicate through outputs and inputs. Module A creates a resource and exposes its attributes via outputs. The root configuration reads Module A’s outputs and passes them as input variables to Module B. Terraform builds a dependency graph from these references and creates resources in the correct order. For example, the storage module outputs the account ID, which the root passes to the RBAC module as the scope for a role assignment.

Q: How would you structure a Terraform CI/CD pipeline in Azure DevOps? A: Separate CI and CD. The CI pipeline runs on every PR: it runs terraform fmt, terraform validate, and terraform plan. The CD pipeline runs on merge to main: Plan Dev → Apply Dev → Plan Prod → Apply Prod. Each plan stage saves the plan to a pipeline artifact. Each apply stage downloads and applies the saved plan. Apply stages use deployment jobs targeting Azure DevOps environments with approval gates. Service principal credentials come from variable groups linked to Key Vault. Templates make the pipeline DRY.

Q: What is the difference between terraform plan -out and terraform apply? A: terraform plan -out=file.tfplan generates an execution plan and saves it to a binary file. terraform apply file.tfplan applies exactly that saved plan — no regeneration, no confirmation prompt. This guarantees the changes you reviewed are the exact changes applied. Without the saved plan, terraform apply regenerates the plan at apply time, which can differ if resources changed between plan and apply. In CI/CD, the plan and apply often run in different pipeline stages minutes apart, making the saved plan essential.

Q: When should you use Terraform vs Databricks Asset Bundles? A: Use Terraform for Azure infrastructure: resource groups, storage accounts, Databricks workspaces, Key Vaults, networking, and Unity Catalog metastore-level objects. Use DABs for Databricks workloads: jobs, notebooks, Lakeflow pipelines, cluster configurations, and catalog-level objects. Terraform manages what the workspace IS. DABs manages what runs INSIDE the workspace. Use both in a complementary CI/CD setup where Terraform creates infrastructure and DABs deploys workloads.

Q: How do you manage Terraform state across multiple environments? A: Use separate state files per environment with different backend configurations. Each environment has its own backend.conf file specifying a unique state key (e.g., dataplatform-dev.tfstate, dataplatform-prod.tfstate). Initialize with terraform init -backend-config=environments/dev/backend.conf. All state files can share the same storage account but use different keys. This isolates environments so a dev apply cannot affect prod state, and team members can work on different environments simultaneously.

Q: What lifecycle rules should you set on production Terraform resources? A: Use lifecycle { prevent_destroy = true } on resources that hold data (storage accounts, databases, Key Vaults) to prevent accidental deletion. Use lifecycle { ignore_changes = [tags] } on resources where tags are managed externally. Use lifecycle { create_before_destroy = true } on resources that cannot have downtime during replacement. These rules act as guardrails that protect against human error in production.

Wrapping Up

Terraform modules transform infrastructure code from flat, repetitive files into organized, reusable, testable components. Combined with Azure DevOps YAML pipelines, they create a complete infrastructure CI/CD pipeline: PR triggers terraform fmt and validate, merge triggers plan and apply through dev and prod with approval gates, and saved plans guarantee what you reviewed is what gets deployed.

The separation between Terraform (infrastructure) and DABs (workloads) keeps each tool doing what it does best. Terraform creates the platform. DABs deploys what runs on it. Both flow through YAML pipelines with templates, variable groups, and approval gates.

Related posts: – Terraform for Data Engineers – YAML Pipelines Deep Dive – Azure DevOps Overview – Databricks Asset Bundles (DABs) – Azure Key Vault

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top