Demo Deploying Loki in Kubernetes

In this lesson you’ll deploy the Grafana + Loki logging stack on a Kubernetes cluster and wire it up with Alloy (agent) for log collection. The cluster for this lab already includes networking and an S3-compatible object store provided by SeaweedFS, which we will use as Loki’s object backend. In production you would typically use a managed S3 or GCS backend. Prerequisites

A running Kubernetes cluster with access via kubectl.
Helm installed and configured to talk to the cluster.
SeaweedFS (S3-compatible) available in the logging namespace in this lab.

Verify the cluster components and running pods:

kubectl get pods -A

Sample output (cluster system components, storage provisioner, SeaweedFS in the logging namespace):

NAMESPACE              NAME                                              READY   STATUS    RESTARTS   AGE
kube-flannel           kube-flannel-ds-x5mvq                             1/1     Running   0          20m
kube-system            coredns-6f6c7df987-5grfx                          1/1     Running   0          20m
kube-system            etcd-controlplane                                 1/1     Running   0          20m
local-path-storage     local-path-provisioner-7f4979bb8b-w5f77           1/1     Running   0          19m
logging                seaweedfs-filer-0                                 1/1     Running   0          19m
logging                seaweedfs-master-0                                1/1     Running   0          19m
logging                seaweedfs-s3-6fd7b5c6f9-qrvxp                     1/1     Running   0          19m
logging                seaweedfs-volume-0                                1/1     Running   0          19m

Overview of what we’ll deploy

Grafana — Visualization UI and data source configuration.
Loki — Log storage and query engine, backed by SeaweedFS (S3-compatible).
Alloy — Agent/DaemonSet to collect pod logs, relabel, and forward to Loki.

Deploy Grafana

Add the Grafana Helm repository and refresh chart indexes:

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update

Create a small values file to expose Grafana via NodePort so it is reachable from the lab environment:

# grafana-values.yaml
service:
  type: NodePort
  nodePort: 32000

Install Grafana into the logging namespace:

helm install grafana grafana-community/grafana --values grafana-values.yaml -n logging

After installation the chart prints useful notes. Retrieve the admin password:

kubectl get secret --namespace logging grafana -o jsonpath='{.data.admin-password}' | base64 --decode ; echo

If you need the NodePort URL to reach Grafana from your workstation:

export NODE_PORT=$(kubectl get --namespace logging -o jsonpath="{.spec.ports[0].nodePort}" services grafana)
export NODE_IP=$(kubectl get nodes --namespace logging -o jsonpath="{.items[0].status.addresses[0].address}")
echo "http://$NODE_IP:$NODE_PORT"

Persistence is disabled by default in this example. You will lose Grafana dashboards and settings if the Grafana pod is deleted or rescheduled unless you enable persistence.

Watch the Grafana pod come up:

watch kubectl get pods -n logging

Find the Loki Helm chart

Locate the Loki chart within the grafana-community repo:

helm search repo loki

You should see grafana-community/loki. Note that older charts included Promtail (a now-deprecated agent in some repos). For current Promtail documentation see: https://grafana.com/docs/loki/latest/clients/promtail/

Install Loki

Create a values file to configure Loki for this lab. It sets:

Single binary deployment mode.
SeaweedFS as an S3-compatible object store.
Authentication disabled for lab convenience.

# loki-values.yaml
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    allow_structured_metadata: true
  pattern_ingester:
    enabled: true
  storage:
    type: s3
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin
  s3:
    endpoint: seaweedfs-s3.logging.svc.cluster.local:8333
    region: us-east-1
    accessKeyId: loki
    secretAccessKey: supersecret1234
    s3ForcePathStyle: true
    insecure: true

deploymentMode: SingleBinary
singleBinary:
  replicas: 1

backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

Install the chart into the logging namespace (pin a chart version if desired):

helm install loki grafana-community/loki --values loki-values.yaml --version 18.4.3 -n logging

Port-forward the Loki gateway for quick local testing:

kubectl port-forward --namespace logging svc/loki-gateway 3100:80 &

Send a test log to Loki (note the required timestamp format in the values array):

curl -H "Content-Type: application/json" -XPOST -s "http://127.0.0.1:3100/loki/api/v1/push" \
  --data-raw '{"streams":[{"stream":{"job":"test"},"values":[["'"$(date +%s)000000000"'", "fizzbuzz"]]}]}'

Query the test log (the label selector uses curly braces; see the example wrapped below):

curl "http://127.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}' | jq .data.result

Watch the Loki pods start:

kubectl get pods -n logging

List services in the logging namespace to identify the loki-gateway service:

kubectl get svc -n logging

Example output (truncated):

NAME                     TYPE        CLUSTER-IP       PORT(S)
grafana                  NodePort    172.20.115.214   80:32000/TCP
loki                     ClusterIP   172.20.24.90     3100/TCP,9095/TCP,9096/TCP
loki-gateway             ClusterIP   172.20.154.105   80/TCP
seaweedfs-s3             ClusterIP   172.20.31.200    8333/TCP,9327/TCP

Connecting Grafana to Loki

Retrieve Grafana admin credentials stored as Kubernetes secrets:

kubectl get secret -n logging grafana -o jsonpath='{.data.admin-user}' | base64 -d
kubectl get secret -n logging grafana -o jsonpath='{.data.admin-password}' | base64 -d

Log into Grafana using the NodePort URL you exported earlier and the admin credentials.

A screenshot of Grafana's dark-themed web UI showing the "Welcome to Grafana" home dashboard with panels for a tutorial, adding a data source, and creating a first dashboard. The left sidebar navigation (Home, Dashboards, Explore, Alerting, Administration, etc.) is visible.

In Grafana:

Click Add data source.
Select “Loki”.
Set the URL to the cluster-local Loki gateway: http://loki-gateway.logging.svc.cluster.local/
Click Save & test — Grafana should show a successful connection.

A Grafana settings screenshot showing the "Connections > Data sources > loki" Additional settings page with fields for HTTP settings (allowed cookies, timeout), alerting toggle, query options, and derived fields. A green banner reads "Data source successfully connected" and action buttons ("Delete", "Remove default", "Save & test") appear at the bottom.

Deploy Alloy as a DaemonSet (log collector)

Alloy discovers pod logs, applies relabeling rules, and forwards logs to Loki. Deploy Alloy as a DaemonSet so an instance runs on every node. Create an Alloy Helm values file that configures it as a DaemonSet and provides the Alloy configuration (applied as a ConfigMap):

# alloy-values.yaml
controller:
  type: daemonset

alloy:
  configMap:
    content: |-
      discovery.kubernetes "pod" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action        = "replace"
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action        = "replace"
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action        = "replace"
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app"]
          action        = "replace"
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets     = discovery.relabel.pod_logs.output
        forward_to  = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push"
        }
      }

Install Alloy into the logging namespace:

helm install alloy grafana-community/alloy --values alloy-values.yaml -n logging

Verify Alloy pods are running (one per node):

kubectl get pods -n logging

Alloy will:

Discover pods via the Kubernetes API.
Apply the relabel rules to produce human-friendly labels: namespace, pod, container, app.
Forward logs to http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push.

Explore logs in Grafana

Open Grafana and go to the Explore tab.
Select the Loki data source you created.
Use label selectors (for example pod=) and the relabeled fields (namespace, pod, container, app) to filter logs.
You should see logs streaming from system and application pods.

Example log line captured by Loki (truncated):

2026-07-29 18:03:01.878 ERROR W0729 15:03:01.878522 1 logging.go:55] [core] [Channel #445] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379"} Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"

Troubleshooting tips

Watch pods and deployments as they start: kubectl get pods -n logging -w.
If pods remain in Pending or ContainerCreating, inspect node resources, taints/tolerations, node selectors, and PVCs.
If Loki clients cannot push logs, validate service reachability from the client namespace: kubectl exec -n logging <pod> -- curl -sS http://loki-gateway.logging.svc.cluster.local/loki/api/v1/health.

Quick reference: deployed components

Component	Purpose	How to access / check
Grafana	Visualization UI and dashboards	NodePort (default lab sample: `http://<NODE_IP>:32000`)
Loki	Log storage & query (TSDB + S3 object store)	`svc/loki` and `svc/loki-gateway` in `logging` namespace
Alloy	DaemonSet log collector, relabeling, forwarder	`kubectl get pods -n logging` shows one per node

Summary

You have deployed:

Grafana for UI and dashboards.
Loki for log ingestion and querying, backed by SeaweedFS as an S3-compatible store in this lab.
Alloy as a DaemonSet to collect, relabel, and forward pod logs to Loki.

Next steps to take in a production-ready environment:

Enable persistence for Grafana (PVCs).
Split Loki into multiple components (ingesters, distributor, querier, ruler) with HA and scaling.
Use a managed S3/GCS object store for durability.
Harden access: enable authentication and secure communication between components.

Links and references

Grafana: https://grafana.com/
Loki documentation: https://grafana.com/oss/loki/
Promtail docs: https://grafana.com/docs/loki/latest/clients/promtail/
SeaweedFS (S3-compatible): https://github.com/chrislusf/seaweedfs
Helm charts — grafana-community: https://grafana-community.github.io/helm-charts/

Course Introduction

Grafana Loki Essentials

Grafana Loki on Kubernetes

Demo Deploying Loki in Kubernetes

Deploy Grafana

Find the Loki Helm chart

Install Loki

Connecting Grafana to Loki

Deploy Alloy as a DaemonSet (log collector)

Explore logs in Grafana

Troubleshooting tips

Summary

Watch Video

Practice Lab

​Deploy Grafana

​Find the Loki Helm chart

​Install Loki

​Connecting Grafana to Loki

​Deploy Alloy as a DaemonSet (log collector)

​Explore logs in Grafana

​Troubleshooting tips

​Summary

Watch Video

Practice Lab

Deploy Grafana

Find the Loki Helm chart

Install Loki

Connecting Grafana to Loki

Deploy Alloy as a DaemonSet (log collector)

Explore logs in Grafana

Troubleshooting tips

Summary