> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Deploying Loki in Kubernetes

> Deploying Grafana, Loki, and Alloy on Kubernetes to collect, store, and visualize pod logs using SeaweedFS S3 compatible storage

In this lesson you'll deploy the Grafana + Loki logging stack on a Kubernetes cluster and wire it up with Alloy (agent) for log collection. The cluster for this lab already includes networking and an S3-compatible object store provided by SeaweedFS, which we will use as Loki's object backend. In production you would typically use a managed S3 or GCS backend.

Prerequisites

* A running Kubernetes cluster with access via `kubectl`.
* Helm installed and configured to talk to the cluster.
* SeaweedFS (S3-compatible) available in the `logging` namespace in this lab.

Verify the cluster components and running pods:

```bash theme={null}
kubectl get pods -A
```

Sample output (cluster system components, storage provisioner, SeaweedFS in the logging namespace):

```console theme={null}
NAMESPACE              NAME                                              READY   STATUS    RESTARTS   AGE
kube-flannel           kube-flannel-ds-x5mvq                             1/1     Running   0          20m
kube-system            coredns-6f6c7df987-5grfx                          1/1     Running   0          20m
kube-system            etcd-controlplane                                 1/1     Running   0          20m
local-path-storage     local-path-provisioner-7f4979bb8b-w5f77           1/1     Running   0          19m
logging                seaweedfs-filer-0                                 1/1     Running   0          19m
logging                seaweedfs-master-0                                1/1     Running   0          19m
logging                seaweedfs-s3-6fd7b5c6f9-qrvxp                     1/1     Running   0          19m
logging                seaweedfs-volume-0                                1/1     Running   0          19m
```

Overview of what we'll deploy

* Grafana — Visualization UI and data source configuration.
* Loki — Log storage and query engine, backed by SeaweedFS (S3-compatible).
* Alloy — Agent/DaemonSet to collect pod logs, relabel, and forward to Loki.

## Deploy Grafana

Add the Grafana Helm repository and refresh chart indexes:

```bash theme={null}
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
```

Create a small values file to expose Grafana via NodePort so it is reachable from the lab environment:

```yaml theme={null}
# grafana-values.yaml
service:
  type: NodePort
  nodePort: 32000
```

Install Grafana into the `logging` namespace:

```bash theme={null}
helm install grafana grafana-community/grafana --values grafana-values.yaml -n logging
```

After installation the chart prints useful notes. Retrieve the admin password:

```bash theme={null}
kubectl get secret --namespace logging grafana -o jsonpath='{.data.admin-password}' | base64 --decode ; echo
```

If you need the NodePort URL to reach Grafana from your workstation:

```bash theme={null}
export NODE_PORT=$(kubectl get --namespace logging -o jsonpath="{.spec.ports[0].nodePort}" services grafana)
export NODE_IP=$(kubectl get nodes --namespace logging -o jsonpath="{.items[0].status.addresses[0].address}")
echo "http://$NODE_IP:$NODE_PORT"
```

Persistence is disabled by default in this example. You will lose Grafana dashboards and settings if the Grafana pod is deleted or rescheduled unless you enable persistence.

Watch the Grafana pod come up:

```bash theme={null}
watch kubectl get pods -n logging
```

## Find the Loki Helm chart

Locate the Loki chart within the grafana-community repo:

```bash theme={null}
helm search repo loki
```

You should see `grafana-community/loki`. Note that older charts included Promtail (a now-deprecated agent in some repos). For current Promtail documentation see: [https://grafana.com/docs/loki/latest/clients/promtail/](https://grafana.com/docs/loki/latest/clients/promtail/)

## Install Loki

Create a values file to configure Loki for this lab. It sets:

* Single binary deployment mode.
* SeaweedFS as an S3-compatible object store.
* Authentication disabled for lab convenience.

```yaml theme={null}
# loki-values.yaml
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    allow_structured_metadata: true
  pattern_ingester:
    enabled: true
  storage:
    type: s3
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin
  s3:
    endpoint: seaweedfs-s3.logging.svc.cluster.local:8333
    region: us-east-1
    accessKeyId: loki
    secretAccessKey: supersecret1234
    s3ForcePathStyle: true
    insecure: true

deploymentMode: SingleBinary
singleBinary:
  replicas: 1

backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0
```

Install the chart into the `logging` namespace (pin a chart version if desired):

```bash theme={null}
helm install loki grafana-community/loki --values loki-values.yaml --version 18.4.3 -n logging
```

Port-forward the Loki gateway for quick local testing:

```bash theme={null}
kubectl port-forward --namespace logging svc/loki-gateway 3100:80 &
```

Send a test log to Loki (note the required timestamp format in the values array):

```bash theme={null}
curl -H "Content-Type: application/json" -XPOST -s "http://127.0.0.1:3100/loki/api/v1/push" \
  --data-raw '{"streams":[{"stream":{"job":"test"},"values":[["'"$(date +%s)000000000"'", "fizzbuzz"]]}]}'
```

Query the test log (the label selector uses curly braces; see the example wrapped below):

```bash theme={null}
curl "http://127.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}' | jq .data.result
```

Watch the Loki pods start:

```bash theme={null}
kubectl get pods -n logging
```

List services in the `logging` namespace to identify the `loki-gateway` service:

```bash theme={null}
kubectl get svc -n logging
```

Example output (truncated):

```console theme={null}
NAME                     TYPE        CLUSTER-IP       PORT(S)
grafana                  NodePort    172.20.115.214   80:32000/TCP
loki                     ClusterIP   172.20.24.90     3100/TCP,9095/TCP,9096/TCP
loki-gateway             ClusterIP   172.20.154.105   80/TCP
seaweedfs-s3             ClusterIP   172.20.31.200    8333/TCP,9327/TCP
```

## Connecting Grafana to Loki

Retrieve Grafana admin credentials stored as Kubernetes secrets:

```bash theme={null}
kubectl get secret -n logging grafana -o jsonpath='{.data.admin-user}' | base64 -d
kubectl get secret -n logging grafana -o jsonpath='{.data.admin-password}' | base64 -d
```

Log into Grafana using the NodePort URL you exported earlier and the admin credentials.

![A screenshot of Grafana's dark-themed web UI showing the "Welcome to Grafana" home dashboard with panels for a tutorial, adding a data source, and creating a first dashboard. The left sidebar navigation (Home, Dashboards, Explore, Alerting, Administration, etc.) is visible.](https://mintcdn.com/kodekloud-c4ac6d9a/dEcIUoAxuol-3ZBK/images/Grafana-Loki/Grafana-Loki-on-Kubernetes/Demo-Deploying-Loki-in-Kubernetes/grafana-welcome-home-dashboard-screenshot.jpg?fit=max\&auto=format\&n=dEcIUoAxuol-3ZBK\&q=85\&s=b736cd99902500e180e1d69513494ab0)

In Grafana:

1. Click Add data source.

2. Select "Loki".

3. Set the URL to the cluster-local Loki gateway:

   `http://loki-gateway.logging.svc.cluster.local/`

4. Click Save & test — Grafana should show a successful connection.

![A Grafana settings screenshot showing the "Connections > Data sources > loki" Additional settings page with fields for HTTP settings (allowed cookies, timeout), alerting toggle, query options, and derived fields. A green banner reads "Data source successfully connected" and action buttons ("Delete", "Remove default", "Save & test") appear at the bottom.](https://mintcdn.com/kodekloud-c4ac6d9a/dEcIUoAxuol-3ZBK/images/Grafana-Loki/Grafana-Loki-on-Kubernetes/Demo-Deploying-Loki-in-Kubernetes/grafana-loki-datasource-settings-connected-banner.jpg?fit=max\&auto=format\&n=dEcIUoAxuol-3ZBK\&q=85\&s=b474f26ff71b5dbda36ad7247952344a)

## Deploy Alloy as a DaemonSet (log collector)

Alloy discovers pod logs, applies relabeling rules, and forwards logs to Loki. Deploy Alloy as a DaemonSet so an instance runs on every node.

Create an Alloy Helm values file that configures it as a DaemonSet and provides the Alloy configuration (applied as a ConfigMap):

```yaml theme={null}
# alloy-values.yaml
controller:
  type: daemonset

alloy:
  configMap:
    content: |-
      discovery.kubernetes "pod" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action        = "replace"
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action        = "replace"
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action        = "replace"
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app"]
          action        = "replace"
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets     = discovery.relabel.pod_logs.output
        forward_to  = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push"
        }
      }
```

Install Alloy into the `logging` namespace:

```bash theme={null}
helm install alloy grafana-community/alloy --values alloy-values.yaml -n logging
```

Verify Alloy pods are running (one per node):

```bash theme={null}
kubectl get pods -n logging
```

Alloy will:

* Discover pods via the Kubernetes API.
* Apply the relabel rules to produce human-friendly labels: `namespace`, `pod`, `container`, `app`.
* Forward logs to `http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push`.

## Explore logs in Grafana

1. Open Grafana and go to the Explore tab.
2. Select the Loki data source you created.
3. Use label selectors (for example `pod=`) and the relabeled fields (`namespace`, `pod`, `container`, `app`) to filter logs.
4. You should see logs streaming from system and application pods.

Example log line captured by Loki (truncated):

```text theme={null}
2026-07-29 18:03:01.878 ERROR W0729 15:03:01.878522 1 logging.go:55] [core] [Channel #445] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379"} Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
```

## Troubleshooting tips

* Watch pods and deployments as they start: `kubectl get pods -n logging -w`.
* If pods remain in `Pending` or `ContainerCreating`, inspect node resources, taints/tolerations, node selectors, and PVCs.
* If Loki clients cannot push logs, validate service reachability from the client namespace: `kubectl exec -n logging <pod> -- curl -sS http://loki-gateway.logging.svc.cluster.local/loki/api/v1/health`.

Quick reference: deployed components

| Component | Purpose | How to access / check |
| - | - | - |
| Grafana | Visualization UI and dashboards | NodePort (default lab sample: `http://<NODE_IP>:32000`) |
| Loki | Log storage & query (TSDB + S3 object store) | `svc/loki` and `svc/loki-gateway` in `logging` namespace |
| Alloy | DaemonSet log collector, relabeling, forwarder | `kubectl get pods -n logging` shows one per node |

## Summary

You have deployed:

* Grafana for UI and dashboards.
* Loki for log ingestion and querying, backed by SeaweedFS as an S3-compatible store in this lab.
* Alloy as a DaemonSet to collect, relabel, and forward pod logs to Loki.

Next steps to take in a production-ready environment:

* Enable persistence for Grafana (PVCs).
* Split Loki into multiple components (ingesters, distributor, querier, ruler) with HA and scaling.
* Use a managed S3/GCS object store for durability.
* Harden access: enable authentication and secure communication between components.

Links and references

* Grafana: [https://grafana.com/](https://grafana.com/)
* Loki documentation: [https://grafana.com/oss/loki/](https://grafana.com/oss/loki/)
* Promtail docs: [https://grafana.com/docs/loki/latest/clients/promtail/](https://grafana.com/docs/loki/latest/clients/promtail/)
* SeaweedFS (S3-compatible): [https://github.com/chrislusf/seaweedfs](https://github.com/chrislusf/seaweedfs)
* Helm charts — grafana-community: [https://grafana-community.github.io/helm-charts/](https://grafana-community.github.io/helm-charts/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/grafana-loki/module/595f27e5-031e-46ea-bffc-ee1a15f73837/lesson/98495d03-5103-45c1-891d-d7a4a01c9c5f)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/grafana-loki/module/595f27e5-031e-46ea-bffc-ee1a15f73837/lesson/0e41d0f1-08f0-44ee-b6dd-dedbcad7a419)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.