Elasticsearch Problem and Solution

Where should Bob’s log lines live so they outlast the pod that wrote them? Bob knows the answer: durable storage and stable identities — Elasticsearch needs both. Below we follow his first attempt (running Elasticsearch like a normal Deployment) and then show the Kubernetes primitives Elasticsearch actually requires to keep data and discovery intact. He runs Elasticsearch the same way he’d run any app — as a Deployment with two replicas. Two pods start, each with a random suffix on the name. One user log line arrives and is written to disk. So far, so good. Then one of those pods is replaced the way the app pod was replaced last night: the Deployment tears down the old pod and starts a fresh one. The new pod gets a brand-new random name and an empty disk because containers write to ephemeral pod-local storage by default. All documents the old pod stored are gone with it. Bob has moved the problem one step down the line: the logs now die with the Elasticsearch pod instead of with the app pod.

A dark infographic titled "One Step Down the Line" shows two smiling blue robot-like icons on separate platforms connected by a right-pointing arrow. The right robot has a "USER1 logged in" label and a small token badge, indicating a login step.

What does Elasticsearch actually need from Kubernetes? Two things a Deployment doesn’t provide:

Per-pod durable storage that outlives the pod so a replacement can pick up exactly where the old one stopped.
A stable, predictable network identity so other Elasticsearch nodes and external clients can reliably find each pod after a restart.

Below are the Kubernetes features that provide those guarantees.

Durable storage: PersistentVolume + PersistentVolumeClaim

Kubernetes separates physical storage from requests for it:

PersistentVolume (PV): the real piece of storage (e.g., a disk, network volume, or a directory on a node).
PersistentVolumeClaim (PVC): a request/ticket a pod uses to mount storage.

Example PV (Bob’s lab uses a node folder for demo purposes):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-elasticsearch
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/elasticsearch

This example uses a hostPath PV for illustration. In production you should use a proper networked storage class or dynamic provisioning so each pod gets its own physical backing. A single ReadWriteOnce PV cannot be mounted by multiple pods simultaneously on different nodes.

Bob doesn’t create PVCs manually for each pod. Instead he puts a claim template inside an Elasticsearch StatefulSet and Kubernetes creates a fresh PVC per pod from that template. Each pod mounts its claim at the Elasticsearch data directory:

volumeMounts:
- name: es-data
  mountPath: /usr/share/elasticsearch/data

volumeClaimTemplates:
- metadata:
    name: es-data
  spec:
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 5Gi

Because each pod gets its own PVC (and that PVC binds to a PV-backed volume), when a pod is replaced the new pod with the same identity can remount the exact same volume and access the data left behind by the old pod.

Stable identity: StatefulSet

Deployments give pods random names and perform rolling updates with interchangeable pods; that’s fine for stateless apps but not for stateful clustered systems that rely on persistent identity. StatefulSet provides:

Stable, ordinal pod names: elasticsearch-0, elasticsearch-1, …
Ordered, deterministic startup and termination.
Automatic creation of per-pod PVCs when volumeClaimTemplates are defined.

With a StatefulSet, when elasticsearch-0 is replaced, the new pod is again elasticsearch-0, mounts the same PVC, and user data remains intact.

A dark-themed diagram titled "Last Night, Replayed" showing an Elasticsearch pod labeled "elasticsearch-0" writing to a data volume "es-data-elasticsearch-0" stored at /data/elasticsearch. A user icon labeled "Bob" and a note "USER1 logged in" are shown to the right.

Stable network names: Services (and headless Service for discovery)

Pod IPs change on restart, so you can’t rely on recorded IP addresses for discovery. Kubernetes Services provide stable DNS names and network endpoints in front of a set of pods. Bob puts one or more Services in front of his Elasticsearch StatefulSet:

A headless Service (clusterIP: None) gives each pod a stable DNS record such as elasticsearch-0.elasticsearch.default.svc.cluster.local, which is useful for Elasticsearch node-to-node discovery.
A ClusterIP/NodePort/LoadBalancer Service provides a stable client endpoint and can load-balance across nodes for client access.

If Bob wants to curl Elasticsearch from a node, he also exposes port 9200 as a NodePort so the HTTP port becomes reachable on the node IP (e.g., node:30200).

A slide titled "A Service in Front" showing Fluentd on the left and a highlighted "elasticsearch" service in front of two app: elasticsearch instances on the right. The diagram labels ports 9200 and 9300 above the service.

Example service ports configuration:

ports:
 - port: 9200
   targetPort: 9200
   nodePort: 30200
   name: http

 - port: 9300
   targetPort: 9300
   nodePort: 30300
   name: transport

type: NodePort

Use a headless Service for internal cluster discovery and a regular Service (ClusterIP/NodePort/LoadBalancer) for client connections when appropriate.

A few other clarifications

Don’t confuse Kubernetes “replicas” with Elasticsearch “replicas”. Kubernetes replica count controls how many pod instances run. Elasticsearch replicas are copies of indices/data across cluster nodes.
For quick testing, run Elasticsearch in single-node mode so it doesn’t attempt cluster formation until you intentionally scale:

discovery.type: single-node
# or in elasticsearch.yml:
# discovery.type: single-node

Quick reference table

Kubernetes resource	Purpose	Example / notes
PersistentVolume (PV)	Actual storage backing (disk, NFS, cloud volume)	See PV example above (`hostPath` used for demo)
PersistentVolumeClaim (PVC)	Request for storage that pods mount	Use `volumeClaimTemplates` in a StatefulSet to auto-create per-pod PVCs
StatefulSet	Stable pod identities and ordered startup	Use for Elasticsearch nodes so each pod name and volume are stable
Service (headless)	Stable DNS records for each pod (discovery)	`clusterIP: None` provides per-pod DNS like `elasticsearch-0.svc`
Service (NodePort/ClusterIP)	Stable endpoint for clients	NodePort exposes a port on each node to reach HTTP port 9200

When to use this pattern

General rule: if losing a pod’s disk loses data you care about, use a StatefulSet with volumeClaimTemplates so each pod has stable identity and storage. If other pods or external clients need to find it by name, put a Service in front of it (headless for discovery; ClusterIP/NodePort/LoadBalancer for clients).

If persistent data matters, use a StatefulSet with volumeClaimTemplates so each pod has a stable identity and stable storage. Use a Service to provide a stable network name (and optionally NodePort) so clients and other nodes can always reach the cluster.

Links and references

Kubernetes Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Kubernetes StatefulSet: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
Kubernetes Services: https://kubernetes.io/docs/concepts/services-networking/service/
Elasticsearch configuration: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

Introduction to EFK on Kubernetes

Deploying Elasticsearch

Collecting Logs With Fluentd

Visualizing Logs With Kibana

Monitoring and Alerting

Scaling the EFK Stack

Securing the EFK Stack

Advanced EFK Configuration

Elasticsearch Problem and Solution

Durable storage: PersistentVolume + PersistentVolumeClaim

Stable identity: StatefulSet

Stable network names: Services (and headless Service for discovery)

A few other clarifications

Quick reference table

When to use this pattern

Links and references

Watch Video

​Durable storage: PersistentVolume + PersistentVolumeClaim

​Stable identity: StatefulSet

​Stable network names: Services (and headless Service for discovery)

​A few other clarifications

​Quick reference table

​When to use this pattern

​Links and references

Watch Video

Durable storage: PersistentVolume + PersistentVolumeClaim

Stable identity: StatefulSet

Stable network names: Services (and headless Service for discovery)

A few other clarifications

Quick reference table

When to use this pattern

Links and references