Designing an Agent Infrastructure

In this article, we explore how to build a robust agent infrastructure for your build agents using both hosted and self-hosted options. We discuss the benefits, configuration details, and various deployment scenarios for Windows, Linux, and Mac agents. This guide is ideal for teams looking to optimize their CI/CD workflows with Azure DevOps.

Self-hosted agents offer greater configurability and cost-effectiveness—especially for high-frequency builds—while hosted agents provide reliable, managed environments maintained by the service provider.

Azure Pipelines Hosted Agent

To begin, navigate to your organization settings in Azure DevOps. In the lower left-hand corner, select Agent Pools to view the different pools you have configured.

For instance, the "Azure Pipelines" pool utilizes a hosted agent. In the interface below, you can see that the hosted agent is actively processing a build:

Hosted build agents support Windows, Linux, and Mac jobs, making them an excellent choice for automated builds that require high availability.

Windows Self-Hosted Agents

When you choose a self-hosted option, you run your build agents on your own infrastructure—either on a local workstation or a dedicated server. In our configuration, we have a separate pool featuring two Windows self-hosted agents.

One of these agents is configured on a machine, as illustrated below:

Starting a Windows Self-Hosted Agent

To bring the agent online, open a Command Prompt, navigate to the agent directory, and execute the command to start it. For example, run the following command in CMD:

C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.CodeReview.Common.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.CodeReview.Discussion.WebApi.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.CodeReview.WebApi.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.Common.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.Services.WebApi.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Microsoft.VisualStudio.VirtualTreeGrid.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\Newtonsoft.Json.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\System.IdentityModel.Tokens.Jwt.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\System.Net.Http.Formatting.dll
VERBOSE: Unblock: C:\Users\jeremy\Downloads\agent\externals\tf\System.Web.Http.dll
Scanning for tool capabilities.

Once executed, the agent will appear online and ready to process jobs. You can manage its status (enable or disable) from the agent pools page.

Containerized (Docker) Windows Agent

For enhanced customization, you can configure a containerized Docker agent. This approach enables you to encapsulate your build environment within a container for consistent and scalable deployments.

First, open the batch file to review or edit its contents:

C:\Users\jeremy\Projects\agents\azpbuild>notepad start.bat

Then, start the Docker container using the following command:

docker run -e AZP_URL="%AZP_URL%" -e AZP_TOKEN="%AZP_TOKEN%" -e AZP_POOL="Default" -e AZP_AGENT_NAME="Docker Agent - Windows" --name "azp-agent-windows" azp-agent

With both the containerized agent and a Windows self-hosted agent in your setup, you gain flexibility for various build scenarios. Check the agent pools in Azure DevOps to verify the status of your agents.

Linux Agents

For Linux builds, you have two primary options: running an agent in Windows Subsystem for Linux (WSL) on your local machine or deploying an agent on a dedicated Linux server.

1. WSL Agent on Local Machine

If you prefer to use WSL, you can quickly launch a Linux agent on your local machine with this script:

jeremy@DIGITALSTORM:~/myagent$ ./run.sh
Scanning for tool capabilities.

This method leverages WSL for building Linux projects efficiently on Windows.

2. Dedicated Linux Server

For high-load or production environments, using a dedicated Linux server is ideal. On an Arch Linux server, start the agent with the following command:

[jeremy@archlinux myagent]$ ./run.sh
Scanning for tool capabilities.
Connecting to the server.
2024-09-26 17:57:45Z: Listening for Jobs

This configuration ensures that resource-intensive builds do not interfere with your local development environment.

Mac Agent

For projects targeting Apple platforms, setting up a Mac agent is essential. In our infrastructure, a Mac Studio handles these tasks. You can easily SSH into the Mac and run your builds as needed.

Choosing the Right Infrastructure

Selecting the appropriate agent infrastructure depends on several factors, including:

• The operating systems you need to support (Linux, macOS, or Windows).
• Whether builds should run on local machines, on dedicated servers, or in the cloud.
• The level of customization required for your builds (e.g., specific software versions or Docker configurations).

For large-scale builds, containerized agents can be scaled across multiple servers or even managed via Kubernetes, addressing both performance and resource optimization.

If you have further questions or require additional guidance on setting up your build agents, feel free to join our forums for community support and discussions.

Thank you for reading this guide on designing an agent infrastructure for efficient CI/CD implementation with Azure DevOps.