diagnostics-integration

name: diagnostics-integration description: Add Azure Monitor diagnostic settings to existing Terraform modules. Use this skill when adding monitoring, logging, or diagnostics capabilities to a module that supports Azure diagnostic settings.

Diagnostics Integration for Azure CAF Modules

Add Azure Monitor diagnostic settings to modules for comprehensive logging and monitoring.

Terraform MCP Integration (Required)

Before adding/changing azurerm_* diagnostics-related arguments, validate with Terraform MCP:

• mcp_terraform_get_provider_details
• mcp_terraform_search_modules + mcp_terraform_get_module_details for reference implementations

When to Use This Skill

Use this skill when:

• Creating a new module for a resource that supports diagnostic settings
• Adding diagnostics to an existing module
• A user asks to "add monitoring" or "enable logging"
• Implementing compliance requirements (e.g., log all operations)

Azure Services That Support Diagnostics

Most Azure services support diagnostic settings. Common examples:

• ✅ Storage Accounts
• ✅ Key Vaults
• ✅ SQL Databases
• ✅ App Services
• ✅ AKS Clusters
• ✅ Azure Cache for Redis
• ✅ Cognitive Services
• ✅ Container Apps
• ✅ Virtual Networks
• ✅ Application Gateways

Verification: Does the Service Support Diagnostics?

Check Azure Provider Documentation:

Resource: azurerm_<resource_name>
Look for: azurerm_monitor_diagnostic_setting referencing this resource type

Or search Azure documentation for "diagnostic settings for [service name]".

Step-by-Step Implementation

Step 1: Check if Module Already Supports Diagnostics

# Check if diagnostics.tf exists
ls modules/<category>/<module_name>/diagnostics.tf

# Check if variable exists
grep "diagnostics" modules/<category>/<module_name>/variables.tf

If the file exists, the module already supports diagnostics. Skip to Step 6 (Creating Examples).

Step 2: Create diagnostics.tf File

Location: modules/<category>/<module_name>/diagnostics.tf

Standard Template:

#
# Diagnostics settings
#

module "diagnostics" {
  source = "../../diagnostics"
  count  = var.diagnostics != null ? 1 : 0

  resource_id       = <resource_reference>.id
  resource_location = try(var.settings.location, local.location)
  diagnostics       = var.diagnostics
  profiles          = try(var.settings.diagnostic_profiles, {})
}

Key Elements:

• Source: Always ../../diagnostics (relative path from module)
• Count: Only create if diagnostics are configured
• resource_id: Reference to the main resource's ID
• resource_location: Use module's location logic
• diagnostics: Pass through diagnostics configuration
• profiles: Optional diagnostic profiles

Examples by Resource Type:

# Storage Account
resource_id = azurerm_storage_account.storage.id

# Key Vault
resource_id = azurerm_key_vault.keyvault.id

# Managed Redis
resource_id = azurerm_redis_cache.redis.id

# Container App
resource_id = azurerm_container_app.app.id

# AKS Cluster
resource_id = azurerm_kubernetes_cluster.aks.id

# Application Gateway
resource_id = azurerm_application_gateway.agw.id

Step 3: Add Diagnostics Variable

Location: modules/<category>/<module_name>/variables.tf

Add this variable:

variable "diagnostics" {
  description = "Diagnostic settings for the resource. Supports Log Analytics, Storage Account, and Event Hub destinations."
  default     = null
}

Standard Pattern:

• Always optional (default = null)
• Generic description mentioning supported destinations
• No type constraint (uses any implicitly)

Step 4: Verify Relative Path

The diagnostics module path must be correct based on module depth.

Standard Module Structure (depth 2):

modules/
└── <category>/
    └── <module_name>/
        └── diagnostics.tf → source = "../../diagnostics"

Module with Submodule (depth 3):

modules/
└── <category>/
    └── <module_name>/
        └── <submodule>/
            └── diagnostics.tf → source = "../../../diagnostics"

Verification Command:

# From module directory
cd modules/<category>/<module_name>/
realpath ../../diagnostics
# Should output: /path/to/repo/modules/diagnostics

Step 5: Update Module Documentation

Add diagnostics information to the module's README.md (if it exists).

Section to Add:

## Diagnostics

This module supports Azure Monitor diagnostic settings for comprehensive logging and monitoring.

### Supported Destinations

- Log Analytics Workspace
- Storage Account
- Event Hub

### Example Configuration

See examples directory for complete diagnostic configurations.

Step 6: Create Example with Diagnostics

Location: examples/<category>/<service>/200-<service>-diagnostics/configuration.tfvars

Standard Example Template:

global_settings = {
  default_region = "region1"
  regions = {
    region1 = "westeurope"
  }
  random_length = 5
}

resource_groups = {
  rg_service = {
    name = "service-test-1"
  }
  rg_monitoring = {
    name = "monitoring-test-1"
  }
}

# Log Analytics for diagnostics
log_analytics_workspaces = {
  law1 = {
    name = "logs-test-1"
    resource_group = {
      key = "rg_monitoring"
    }
    sku = "PerGB2018"
    retention_in_days = 30
  }
}

# Storage Account for diagnostics (optional)
storage_accounts = {
  diag_storage = {
    name = "diaglogs"
    resource_group = {
      key = "rg_monitoring"
    }
    account_tier             = "Standard"
    account_replication_type = "LRS"
  }
}

# Main service with diagnostics
<category> = {
  <service_name> = {
    service1 = {
      name = "service-instance-1"
      resource_group = {
        key = "rg_service"
      }
      
      # Service-specific configuration
      # ... (service settings)
      
      # Diagnostics configuration
      diagnostic_profiles = {
        operations = {
          name = "operations_logs"
          
          # Logs configuration
          log = [
            {
              category = "Administrative"
              enabled  = true
              retention_policy = {
                enabled = true
                days    = 90
              }
            },
            {
              category = "Security"
              enabled  = true
              retention_policy = {
                enabled = true
                days    = 365
              }
            }
          ]
          
          # Metrics configuration
          metric = [
            {
              category = "AllMetrics"
              enabled  = true
              retention_policy = {
                enabled = true
                days    = 30
              }
            }
          ]
          
          # Destinations
          log_analytics_destination_type = "Dedicated"
        }
      }
    }
  }
}

# Diagnostics object (passed to module)
diagnostics = {
  operations = {
    log_analytics = {
      log_analytics_workspace = {
        key = "law1"
      }
    }
    storage_account = {
      storage_account = {
        key = "diag_storage"
      }
    }
  }
}

Key Components:

1. Log Analytics Workspace - Primary destination
2. Storage Account - Optional long-term storage
3. Diagnostic Profiles - Define what to log
4. Diagnostics Object - Maps profiles to destinations

Step 7: Common Log Categories by Service

Different Azure services support different log categories. Here are common patterns:

Storage Account:

log = [
  { category = "StorageRead" },
  { category = "StorageWrite" },
  { category = "StorageDelete" }
]

Key Vault:

log = [
  { category = "AuditEvent" },
  { category = "AzurePolicyEvaluationDetails" }
]

AKS Cluster:

log = [
  { category = "kube-apiserver" },
  { category = "kube-controller-manager" },
  { category = "kube-scheduler" },
  { category = "kube-audit" },
  { category = "cluster-autoscaler" }
]

App Service:

log = [
  { category = "AppServiceHTTPLogs" },
  { category = "AppServiceConsoleLogs" },
  { category = "AppServiceAppLogs" },
  { category = "AppServiceAuditLogs" }
]

SQL Database:

log = [
  { category = "SQLInsights" },
  { category = "AutomaticTuning" },
  { category = "QueryStoreRuntimeStatistics" },
  { category = "Errors" },
  { category = "DatabaseWaitStatistics" }
]

Consult Azure documentation for complete list per service.

Step 8: Test Diagnostics with Mock Tests

Note: Mock tests use the same configuration file as deployment examples. No separate mock files needed.

Step 9: Test Diagnostics Integration

Mock Test:

cd examples
terraform test \
  -test-directory=./tests/mock \
  -var-file=./<category>/<service>/200-<service>-diagnostics/configuration.tfvars \
  -verbose

Alternative (single command):

terraform -chdir=examples test \
  -test-directory=./tests/mock \
  -var-file=./<category>/<service>/200-<service>-diagnostics/configuration.tfvars \
  -verbose

Real Deployment Test (optional):

# Verify Azure subscription first
az account show --query "{subscriptionId:id, name:name}" -o table
export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)

cd examples
terraform_with_var_files \
  --dir /<category>/<service>/200-<service>-diagnostics/ \
  --action plan \
  --auto auto \
  --workspace test

Step 10: Add to CI/CD (Optional)

If creating a comprehensive diagnostics example, add to workflow:

Location: .github/workflows/standalone-scenarios*.json

{
  "config_files": [
    "existing/examples",
    "<category>/<service>/200-<service>-diagnostics",
    "more/examples"
  ]
}

Diagnostics Module Structure

The shared diagnostics module (modules/diagnostics/) handles all diagnostic settings uniformly.

What it does:

• Creates azurerm_monitor_diagnostic_setting resource
• Resolves Log Analytics workspace by key or ID
• Resolves Storage Account by key or ID
• Resolves Event Hub by key or ID
• Applies diagnostic profiles
• Manages log categories and metrics

You don't need to modify the diagnostics module itself - just use it.

Integration Checklist

• Service supports diagnostic settings (verified in Azure docs)
• Created diagnostics.tf in module directory
• Added diagnostics variable to variables.tf
• Verified relative path to diagnostics module (../../diagnostics)
• Created deployment example (200-level) with diagnostics
• Tested mock test passes
• Updated module README with diagnostics info (optional)
• Added to CI/CD workflow (optional)

Common Issues and Solutions

Diagnostics Best Practices

1. Retention Policies

# Security logs - long retention
retention_policy = {
  enabled = true
  days    = 365
}

# Operational logs - medium retention
retention_policy = {
  enabled = true
  days    = 90
}

# Debug logs - short retention
retention_policy = {
  enabled = true
  days    = 30
}

2. Essential Categories

Always enable these if available:

• Administrative operations
• Security events
• Audit logs
• AllMetrics (for performance monitoring)

3. Multiple Destinations

diagnostics = {
  operations = {
    # Hot path - Log Analytics for queries
    log_analytics = {
      log_analytics_workspace = { key = "law1" }
    }
    # Cold path - Storage for long-term retention
    storage_account = {
      storage_account = { key = "archive_storage" }
    }
    # Event-driven - Event Hub for SIEM integration
    event_hub = {
      event_hub_namespace = { key = "eventhub1" }
      event_hub_name = "security-events"
    }
  }
}

4. Dedicated Log Analytics Tables

log_analytics_destination_type = "Dedicated"  # Creates resource-specific tables
# vs
log_analytics_destination_type = "AzureDiagnostics"  # Uses shared table (default)

Use Dedicated for:

• Better query performance
• Separate retention policies
• Easier data management

Example Services with Diagnostics

Reference these existing implementations:

# Find modules with diagnostics already implemented
find modules -name "diagnostics.tf" -type f

# Example outputs:
# modules/storage/storage_account/diagnostics.tf
# modules/keyvault/keyvault/diagnostics.tf
# modules/databases/mssql_database/diagnostics.tf
# modules/networking/application_gateway/diagnostics.tf

Study existing patterns:

cat modules/storage/storage_account/diagnostics.tf
cat examples/storage/storage_account/200-storage-account-diagnostics/configuration.tfvars

Quick Reference: Standard Diagnostics Block

In diagnostics.tf:

module "diagnostics" {
  source            = "../../diagnostics"
  count             = var.diagnostics != null ? 1 : 0
  resource_id       = <resource>.id
  resource_location = local.location
  diagnostics       = var.diagnostics
  profiles          = try(var.settings.diagnostic_profiles, {})
}

In variables.tf:

variable "diagnostics" {
  description = "Diagnostic settings for the resource."
  default     = null
}

In examples:

diagnostics = {
  profile_name = {
    log_analytics = {
      log_analytics_workspace = { key = "law1" }
    }
  }
}

Related Skills

• module-creation - Create module with diagnostics from start
• mock-testing - Test diagnostics integration
• example-creation - Create comprehensive examples

Summary

Adding diagnostics to a module:

1. ✅ Create diagnostics.tf with standard module call
2. ✅ Add diagnostics variable
3. ✅ Verify relative path
4. ✅ Create examples with diagnostics configuration
5. ✅ Test with mock tests
6. ✅ Document in README

This enables comprehensive Azure Monitor integration for logging, metrics, and compliance.