Viewing Kubernetes logs

In this guide, we demonstrate how to view Kubernetes logs and review the default Promtail configurations used in a Loki-Grafana environment. You'll learn how to query logs in Grafana, inspect the Promtail configuration, and understand how Kubernetes service discovery and relabeling work together to provide an efficient logging solution.

Querying Pod Logs in Grafana

After navigating to the Explore section in Grafana, run a query to confirm that logs are being collected. When selecting a label (for example, "cd mini") and executing a query, you'll see various system labels—such as app, component, file name, instance, job, and namespace—that are automatically assigned to system pods (e.g., etcd server, kube API, kube controller) as well as to Loki, Grafana, and Promtail pods.

For example, to view logs for a specific pod (e.g., the etcd pod in the "minikube" namespace), use the following query:

{pod="etcd-minikube"} |= ``

This query returns all logs generated by the etcd pod, along with metadata labels such as node name, scrape job (defined within the Promtail configuration), and the source file path (typically located under /var/log/pods). You can further refine your queries by modifying or adding extra labels. For instance:

{pod="etcd-minikube"} |= ''

This setup illustrates how logs from containers and pods in a Kubernetes environment are collected and transmitted to Grafana via Loki and Promtail.

Reviewing Promtail Configurations

Next, we examine the Promtail configuration to understand how logs are discovered, processed, and transmitted.

Retrieving the Loki-Grafana Admin Password

First, verify the presence of the Loki-Grafana secret and retrieve the admin password:

kubectl get secret loki-grafana -o jsonpath="{.data.admin-password}"
Error from server (NotFound): secrets "loki-grafana" not found
kubectl get secret loki-grafana -o jsonpath="{.data.admin-password}"
VH5YTD0UdVkVHRXVuZCVJQUSoThqZ1g2QnlxblM3cUZRjV5Tg==
kubectl get secret loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode
Tqya7NQdWDGUuvqRANHE8jXByqan5g7FkN5yN

Checking Running Pods and Secrets

Ensure that your Loki stack is running by listing the current pods:

kubectl get pod
NAME                                   READY   STATUS    RESTARTS   AGE
loki-0                                 1/1     Running   0          10m
loki-grafana-5df4f4fd99-x4794         2/2     Running   0          10m
loki-promtail-bk9rj                    1/1     Running   0          10m
kubectl get sec

Then, inspect the Promtail pod to examine mounts and container configuration. The Promtail container uses several volumes, including a configuration volume mounted from a secret and directories for Docker containers and pod logs.

Below is an excerpt from the Promtail pod configuration:

IP: 172.17.0.3
Controlled By: DaemonSet/loki-promtail
Containers:
  promtail:
    Container ID: docker://45bf8093b9c0cc175903ad18322dbfc52107e3f115ef2473913f6ab89eec576c
    Image: docker.io/grafana/promtail:2.8.3
    Image ID: docker-pullable://grafana/promtail@sha256:48e658500dd2107c1fa08c674472ad199b8dec220c1283fb6c9b990c49ae6f0
    Port: 3101/TCP
    Host Port: 0/TCP
    Args:
      -config.file=/etc/promtail/promtail.yaml
    State: Running
    Started: Sun, 06 Aug 2023 19:04:05 -0400
    Ready: True
    Restart Count: 0
    Readiness: http-get http://http-metrics/ready delay=10s timeout=1s period=10s #success=1 #failure=5
    Environment:
      HOSTNAME: (v1:spec.nodeName)
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s9fm7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True

This Promtail container configuration is defined by a Kubernetes secret (named loki-promtail) that supplies the Promtail configuration file.

Inspecting the Volume Configuration

Review the volume configuration, which shows how various directories are mounted within the Promtail pod:

Conditions:
  Type: Status
  Initialized: True
  Ready: True
  ContainersReady: True
  PodScheduled: True
Volumes:
  config:
    Type: Secret (a volume populated by a Secret)
    SecretName: loki-promtail
    Optional: false
  run:
    Type: HostPath (bare host directory volume)
    Path: /run/promtail
    HostPathType: ""
  containers:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/docker/containers
    HostPathType: ""
  pods:
    Type: HostPath (bare host directory volume)
    Path: /var/log/pods
    HostPathType: ""
  kube-api-access-s9fm7:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional: <nil>
    DownwardAPI: true

Decoding the Promtail Configuration

To inspect the actual Promtail configuration, retrieve and decode the secret:

kubectl get secret loki-promtail -o jsonpath="{.data.promtail\.yaml}"

Then decode the secret’s content using base64. The decoded configuration outlines both client and scrape settings. For example, a snippet of the configuration is:

- __meta_kubernetes_pod_container_name:
    target_label: container
- action: replace
  replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
    - __meta_kubernetes_pod_uid
    - __meta_kubernetes_pod_container_name
  target_label: __path__
- action: replace
  regex: true/(.*)
  replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
    - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
    - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
    - __meta_kubernetes_pod_container_name
  target_label: __path__

limits_config:
tracing:
  enabled: false

This configuration instructs Promtail on where to locate log files (typically under /var/log/pods) and how to assign labels based on the pod’s metadata.

Understanding the Scrape Configuration

The scrape configuration uses Kubernetes service discovery to locate all pods and applies relabeling rules to standardize log labels. An excerpt of the scrape configuration follows:

- cri: {}
kubernetes_sd_configs:
  - role: pod
relabel_configs:
  - source_labels:
      - __meta_kubernetes_pod_controller_name
    regex: ([0-9a-z-]{8,10})?
    action: replace
    target_label: __tmp_controller_name
  - source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_name
      - __meta_kubernetes_pod_label_app
      - __tmp_controller_name
      - __meta_kubernetes_pod_name
    regex: ^*([^\-]+)(;.*)?$
    action: replace
    target_label: app
  - source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_instance
      - __meta_kubernetes_pod_label_instance
    regex: ^*([^\-]+)(;.*)?$
    action: replace
    target_label: instance
  - source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_component
      - __meta_kubernetes_pod_label_component
    regex: ^*([^\-]+)(;.*)?$
    action: replace
    target_label: component

These relabeling rules ensure that critical metadata is preserved in the logs, making it easier to query and manage log data effectively in Grafana.

By following these steps and reviewing the configurations, you can monitor your Kubernetes environment's logs with Loki, Promtail, and Grafana effectively.

Additional Resources

• Kubernetes Documentation
• Grafana Loki Documentation
• Promtail Documentation

Happy logging!