Exploring Azure Pipelines Secrets with Azure Key Vault

In this lesson, learn how to securely manage secrets in your CI/CD pipelines by integrating Azure Key Vault with Azure DevOps. This guide details the step-by-step process of setting up an Azure Key Vault, adding secrets, and accessing them within an Azure Pipeline. Mastering this integration is essential for maintaining robust security and compliance in your deployment workflows.

1. Creating and Configuring an Azure Key Vault

Start by signing in to the Azure Portal and searching for "Key Vaults". Click Create and follow these steps:

1. Choose your subscription and select a resource group.
2. Enter a unique Key Vault name (for example, KodeKloudKeyVault123). The name must be unique across all of Azure.
3. Select a region (we’re using East US) and choose the Standard pricing tier.

By default, soft delete is enabled so that deleted key vaults are retained for up to 90 days. You can modify the retention period or enable/disable purge protection based on your requirements. Click Next to continue.

Next, select the Vault Access Policy permission model. Although Azure Role-Based Access Control is recommended for many scenarios, the Vault Access Policy model provides fine-grained control, simplifying integration with tools like Azure Pipelines.

Review all settings and click Create to deploy your new Key Vault. Once deployment is complete, click Go to Resource to open your Key Vault.

2. Adding a Secret to the Key Vault

Follow these steps to add a secret:

1. In your Key Vault, navigate to Access Policies on the left-hand menu to verify you have the necessary permissions (get, list, create, import, and delete).
2. Select Secrets, then click Generate/Import to add a new secret.
3. Enter a name (e.g., DBPassword) and its value (e.g., 123). Optionally, configure activation and expiration dates to control the secret's validity.

Click Create. Once the secret is added, note the Key Vault name (KodeKloudKeyVault123) and secret name (DBPassword).

3. Setting Up the Azure Service Connection

Integrate the Key Vault with your Azure DevOps project by following these steps:

1. Navigate to Project Settings > Pipelines > Service connections.
2. Click Create Service Connection and choose Azure Resource Manager.
3. Select Service Principal (automatic) at the Subscription Level and choose your subscription and resource group (e.g., AZ-400-DevOps).
4. Name the service connection (e.g., KodeKloud Key Vault Connection) and enable Grant access permission to all pipelines.

4. Building and Configuring a Pipeline

With the Key Vault and service connection configured, create a new pipeline in Azure DevOps. Attach the pipeline to your repository and start with a minimal starter pipeline. Initially, set the trigger to none to allow manual execution. The following is a sample starter pipeline:

# Starter pipeline
trigger:
- none

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

After confirming that the pipeline runs successfully and verifying that the build agents are operational, modify the pipeline to integrate Key Vault secret retrieval.

4.1 Linking a Variable Group for Key Vault Secrets

Link your Azure Key Vault secrets to the pipeline by creating a variable group in the Pipelines Library:

1. Create a variable group named KeyVaultSecrets.
2. Link this group to your Azure Key Vault by selecting the service connection (e.g., KodeKloud Key Vault Connection).
3. Authorize the connection and add the required secret (e.g., DBPassword) to the group.

5. Using the Secret in a Pipeline

Below is an example YAML pipeline that retrieves a secret from Azure Key Vault and uses it within a script. Remember, secrets are automatically masked in the logs:

pool:
  vmImage: ubuntu-latest

variables:
  - group: KeyVaultSecrets

steps:
  - script: echo Hello, world!
    displayName: 'Run a one-line script'

  - task: AzureKeyVault@2
    inputs:
      azureSubscription: 'KodeKloudKeyVaultConnection'
      KeyVaultName: 'KodeKloudKeyVault123'
      SecretsFilter: 'DBPassword'
      RunAsPreJob: false

  - script: |
      echo "Using database password: $(DBPassword)"
    displayName: 'Use the secret'

When you run the pipeline, you might be prompted to grant permissions for the pipeline to access the Key Vault secrets. Once approved, the pipeline retrieves the secret and uses it in the script step. Although the secret's actual value isn’t visible in the logs, its successful retrieval is confirmed through the execution output.

Below is a snippet showing parts of the pipeline log with key details:

# Job preparation parameters example snippet
Variables:
  DBPassword: ${{ variables['KeyVaultSecrets.DBPassword'] }}
  ContinueOnError: False
  TimeoutInMinutes: 60
  CancelTimeoutInMinutes: 5

6. Additional Examples

Below are some examples of integrating Azure Key Vault secrets in different deployment scenarios:

Integrating with .NET Core

Use a .NET Core CLI task to build your project while securely fetching a connection string from Azure Key Vault:

steps:
- task: AzureKeyVault@2
  inputs:
    azureSubscription: 'MyKeyVaultConnection'
    KeyVaultName: 'MyDemoKeyVault'
    SecretsFilter: 'DatabasePassword'
- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
    projects: '**/*.csproj'
  env:
    ConnectionString: "Server=myserver;Database=mydb;User Id=admin;Password=$(DatabasePassword)"

Deploying to an Azure Web App

Securely deploy an Azure Web App by retrieving an API key stored in the Key Vault:

steps:
- task: AzureKeyVault@2
  inputs:
    azureSubscription: 'MyKeyVaultConnection'
    KeyVaultName: 'MyDemoKeyVault'
    SecretsFilter: 'ApiKey'
- task: AzureWebApp@1
  inputs:
    azureSubscription: 'MyAzureSubscription'
    appName: 'MyWebApp'
    deployToSlotOrASE: true
    resourceGroupName: 'MyResourceGroup'
    slotName: 'production'
    appSettings: '-ApiKey $(ApiKey)'

7. Best Practices for Managing Secrets

By following these detailed steps and best practices, you can securely integrate Azure Key Vault with your Azure Pipelines, ensuring your sensitive information remains protected while enhancing your deployment workflows.

Happy coding and secure deployments!