Monitoring Kubernetes Cluster using in Elastic Agent

Welcome to this comprehensive guide on monitoring your Kubernetes cluster with Elastic Agent. In this tutorial, you will learn how to configure Elastic Agent on your Kubernetes cluster, forward logs and metrics to Elastic Cloud, and visualize your data on a pre-configured dashboard.

Step 1: Add the Kubernetes Integration

After logging into your Elastic Cloud console, click on the Add integration option. You will be presented with a list of available integrations:

The image shows a webpage from Elastic, displaying a list of integrations for various Amazon Web Services (AWS) and other categories. The left sidebar lists categories, while the main section shows specific AWS services with descriptions.

From the list, choose the Kubernetes integration and then click on Add Kubernetes. The console provides you with an installation script that will deploy the Elastic Agent on your Kubernetes cluster.

The image is a guide for adding an integration in Elastic, showing three steps: installing the Elastic Agent, adding the integration, and confirming incoming data.

Step 2: Deploy the Elastic Agent on Kubernetes

Click the Install Elastic Agent option to view the Kubernetes deployment YAML along with the command required for deployment. The top portion of the deployment file appears as follows:

---
# For more information see: https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-managed-by-fleet.html
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:

Below is the command to deploy the Elastic Agent:

kubectl apply -f elastic-agent-managed-kubernetes.yml

Next, copy the complete YAML content from the page into a new file (e.g., elastic-agent-managed-kubernetes.yml). Ensure that your YAML file includes the necessary specifications for proper scheduling, such as tolerations for control-plane nodes, to ensure that metrics from Kubernetes control components (scheduler, controller manager) are collected:

spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
  spec:
    # Tolerations are needed to run Elastic Agent on Kubernetes control-plane nodes.
    # Agents running on control-plane nodes collect metrics from the control plane
    # components (scheduler, controller manager) of Kubernetes.
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
    serviceAccountName: elastic-agent

Apply the updated configuration with:

kubectl apply -f elastic-agent-managed-kubernetes.yml

Step 3: Verify the Deployment

Once the deployment process is complete, verify that the Elastic Agent pods are running. Execute the following command:

kubectl get pods -n kube-system

You should see output similar to this:

NAME                                      READY   STATUS              RESTARTS   AGE
coredns-64d57b6bd-6nfr4                   1/1     Running             0          8m37s
coredns-64d57b6bd-h8742                   1/1     Running             0          8m37s
elastic-agent-mxmx4                       0/1     ContainerCreating   0          17s
elastic-agent-npl15                       0/1     ContainerCreating   0          17s
etcd-controlplane                         1/1     Running             0          8m50s
kube-apiserver-controlplane               1/1     Running             0          8m48s
kube-controller-manager-controlplane      1/1     Running             0          8m24s
kube-flannel-ds-t8w7                       1/1     Running             0          8m37s
kube-proxy-dt7jg                          1/1     Running             0          8m37s
kube-scheduler-controlplane               1/1     Running             0          8m48s

After a short wait, the Elastic Agent pods will transition to the running state. Re-run the command above to ensure that both agents are active. You should also see console output similar to the following when the daemonset deployment is applied:

daemonset.apps/elastic-agent created
clusterrolebinding.rbac.authorization.k8s.io/elastic-agent created
rolebinding.rbac.authorization.k8s.io/elastic-agent created
clusterrole.rbac.authorization.k8s.io/elastic-agent created
role.rbac.authorization.k8s.io/elastic-agent created
serviceaccount/elastic-agent created

Step 4: View Metrics in Elastic Cloud

Once the Elastic Agent is enrolled, Elastic Cloud automatically starts collecting metrics from your Kubernetes cluster. In your Elastic Cloud console:

Click on Confirm data.
Select View Kubernetes metrics on the dashboard.

The image shows a preview of incoming data logs in an Elastic interface, detailing various agent activities and configurations. It includes timestamps, agent names, types, versions, and cloud instance information.

Within a few minutes, you will see metrics such as node details, memory usage, CPU core counts, and even information about the top memory-intensive pods appear on the dashboard. For instance, the default dashboard displays the Elastic Agent running in the kube-system namespace:

The image shows a dashboard from Elastic displaying Kubernetes metrics, including memory usage and pod information, with a filter option for namespaces.

If you require a customized view, you can clone or modify the dashboard. However, note that managed dashboards provided by Elastic Cloud are maintained by the Elastic Agent and are not immediately editable.

Step 5: Clean Up

If you’re using a free trial Elastic Cloud account, be sure to delete the deployment after your trial period ends to prevent ongoing charges.

To delete your deployment:

Navigate to your profile in Elastic Cloud.
Click on Organization, then Elastic.
On the home page, click Manage and choose Delete deployment.
Confirm deletion by typing the cluster name.

This ensures that you will not incur any unwanted costs after your trial period.

Conclusion

This guide demonstrated how to seamlessly monitor your Kubernetes cluster with Elastic Agent, leveraging the robust observability and logging capabilities of Elastic Cloud. For further reading, explore additional resources:

Thank you for following this guide. Happy monitoring, and we look forward to bringing you more insightful lessons in the future!

Mastering Elasticsearch Fundamentals

What Is Logging and Elasticsearch?

Understanding Kibana

Fluent Bit

Elasticsearch and Kibana Deployment on Kubernetes

Deploying E-Commerce Application on K8s

Instrumenting a Simple Python App for Logging

Elastic Cloud

Introduction

Monitoring Kubernetes Cluster using in Elastic Agent

Step 1: Add the Kubernetes Integration

Step 2: Deploy the Elastic Agent on Kubernetes

Step 3: Verify the Deployment

Step 4: View Metrics in Elastic Cloud

Step 5: Clean Up

Conclusion

Watch Video

Practice Lab

​Step 1: Add the Kubernetes Integration

​Step 2: Deploy the Elastic Agent on Kubernetes

​Step 3: Verify the Deployment

​Step 4: View Metrics in Elastic Cloud

​Step 5: Clean Up

​Conclusion

Watch Video

Practice Lab

Step 1: Add the Kubernetes Integration

Step 2: Deploy the Elastic Agent on Kubernetes

Step 3: Verify the Deployment

Step 4: View Metrics in Elastic Cloud

Step 5: Clean Up

Conclusion