Demo Configure cloud instances Kubernetes

In this guide, you’ll learn how to integrate a Kubernetes cluster with Jenkins to provision dynamic build agents. We’ll cover plugin installation, cloud setup using both a full-admin kubeconfig and a least-privileged service account, and finalize key settings like pod retention and labels.

1. Install the Kubernetes Plugin

Jenkins requires the Kubernetes plugin to spin up agents in your cluster.

Installation Method	Command / Steps
Jenkins UI	Manage Jenkins → Manage Plugins → Available → Filter “Cloud” → Select Kubernetes 4.2.9.5 → Install
CLI	`jenkins-plugin-cli --plugins kubernetes:4.2.9.5`
Advanced (HPI upload)	Upload from URL: `https://updates.jenkins.io/download/plugins/kubernetes/4.2.9.5/kubernetes.hpi`

The image shows a Jenkins plugin management interface with a list of available plugins related to cloud providers, such as Docker, Kubernetes, and Amazon EC2.

If dependency errors occur (e.g., an out-of-date Credentials plugin), update those first and restart Jenkins:

The image shows a Jenkins interface displaying the download progress of plugins, with a failure message related to the Kubernetes Credentials Plugin. It indicates a need for a plugin update and a Jenkins restart.

Note

Always restart Jenkins after plugin upgrades to ensure dependencies load correctly.

2. Configure the Kubernetes Cloud

Go to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds.
Click Add a new cloud and select Kubernetes.
Provide a name (e.g., prod-k8s-us-east).

You’ll see the Kubernetes cloud configuration form:

The image shows a Jenkins configuration page for setting up a new cloud, with options for Kubernetes Namespace, Agent Docker Registry, and connection settings.

A. Connect Using a Kubeconfig File

Export your full kubeconfig:

kubectl config view --raw > kubeconfig.yaml

In Jenkins, add a Secret file credential (kubeconfig-us-east) and upload kubeconfig.yaml.
Select this credential under Kubernetes Credentials and click Test Connection.

A successful connection shows your cluster version:

The image shows a Jenkins configuration screen for setting up a new cloud, with options for Kubernetes namespace, agent Docker registry, and credentials. It indicates a connection to Kubernetes version 1.29.9.

Note

Using a full-admin kubeconfig grants access to the entire cluster. For production, it’s best practice to use a least-privileged service account.

B. Connect Using a Service Account Token

Follow these steps to lock down permissions:

Create a namespace and service account:

kubectl create namespace jenkins
kubectl -n jenkins create serviceaccount jenkins-sa

Generate a long-lived token:

kubectl -n jenkins create token jenkins-sa --duration=115d

In Jenkins, add a Secret text credential (jenkins-sa-token) with this token.
Back in the Kubernetes cloud config:
- Kubernetes URL: your API server endpoint
- Namespace: jenkins
- Credentials: jenkins-sa-token
- Click Test Connection.

Initially, you may encounter a certificate path error:

The image shows a Jenkins configuration page with an error message indicating a connection test failure due to a certification path issue. The interface includes options for adding credentials, testing connections, and setting URLs.

Warning

Disabling TLS verification is insecure. Instead, provide the CA certificate for your API server under Kubernetes CA Certificate.

You may then hit a 403 Forbidden error due to missing RBAC permissions:

The image shows a Jenkins configuration page for setting up a new cloud, with fields for Kubernetes Namespace and Agent Docker Registry. There is an error message indicating a failure to list resources in the specified namespace due to permission issues.

Grant namespace-scoped admin rights:

kubectl -n jenkins create rolebinding jenkins-admin-binding \
  --clusterrole=admin \
  --serviceaccount=jenkins:jenkins-sa

In Jenkins, click Test Connection again. You should see the credentials dropdown populated and a successful response:

The image shows a Jenkins configuration screen for creating a new cloud, with options for Kubernetes namespace, agent Docker registry, and a dropdown menu for selecting credentials.

3. Finalize Cloud Settings

Configure how Jenkins launches and cleans up pods:

Jenkins URL / Jenkins tunnel: Host:port for JNLP/WebSocket agent connections.
Pod Labels: e.g., organization=KodeKloudAzureArc—tags applied to every agent pod.
Pod Retention:

Option	Description
Never	Delete pods immediately after build
On failure	Keep pods only if the build fails
Always	Retain pods regardless of build outcome

The image shows a Jenkins configuration page for setting up a new cloud, with fields for WebSocket, Jenkins URL, Jenkins tunnel, connection timeout, read timeout, concurrency limit, and pod labels.

Click Save. Jenkins will now provision build agents dynamically in your Kubernetes cluster!

References

Watch Video

Watch video content