Hands On Security

Which lock goes on first? Elasticsearch — because it’s the vault everything else talks to. And how does Bob prove each lock took hold? The same way he proved scaling worked: after every change, he reads something back from the cluster — the actual answer, not just a feeling that a file looked right. For example, when he queries Elasticsearch he expects a response like:

{
  "name": "elasticsearch-0"
}

Bob opens the Elasticsearch UI on port 30200 in his browser, and the details of his Elasticsearch fill the page for anyone who asks:

30200
"name" : "elasticsearch-0"
"cluster_name" : "docker-cluster"
"number" : "7.1.0"

Time to change that.

Enable X-Pack security and TLS for Elasticsearch

Bob edits the Elasticsearch StatefulSet and adds environment variables that enable X-Pack security and TLS for transport (node-to-node) communication:

env:
  - name: discovery.type
    value: single-node
  - name: xpack.security.enabled
    value: "true"
  - name: xpack.security.transport.ssl.enabled
    value: "true"

The examples here use plain environment entries in the StatefulSet for clarity. In production, use ConfigMaps for non-sensitive configuration and Secrets for passwords; do not store credentials directly in manifests. Also note Kubernetes environment variable names must follow naming rules (some API versions disallow dots). How you pass these settings depends on your Elasticsearch image and deployment method.

A further note: setting xpack.security.transport.ssl.enabled=true secures the Elasticsearch transport layer (node-to-node). If you want the HTTP/REST API (port 9200) to use HTTPS so clients can verify certificates, enable xpack.security.http.ssl.enabled and configure certificates for the HTTP layer; otherwise HTTP remains plain and client-side TLS verification will not apply. Bob applies the updated StatefulSet and Kubernetes replaces the Elasticsearch pods with new ones that include the settings.

$ kubectl apply -f es-statefulset.yaml
statefulset.apps/elasticsearch configured

When Bob refreshes the Elasticsearch UI, the page no longer exposes cluster details — instead it asks for a username and password. But which username? There isn’t one yet.

Create a superuser inside the Elasticsearch pod

Bob steps into the running Elasticsearch pod and uses the elasticsearch-users tool that ships in the image to add a superuser account. He runs a single command to create elastic_user with a password:

$ kubectl exec -it elasticsearch-0 -- \
  bin/elasticsearch-users useradd elastic_user -r superuser -p elasticPass123

Now he checks by calling Elasticsearch on the node port with those credentials; Elasticsearch answers when authenticated:

$ curl -u elastic_user:elasticPass123 localhost:30200
{
  "name" : "elasticsearch-0"
}

Without credentials the same endpoint returns a security error:

$ curl localhost:30200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token"}],"type":"security_exception","reason":"missing authentication token"},"status":401}

Kibana must authenticate to Elasticsearch

Now, Bob opens Kibana — and it’s broken. The Kibana UI reports it cannot connect to Elasticsearch. Why? Because Kibana is just another client, and it’s still trying to talk to Elasticsearch without credentials. Bob edits the Kibana Deployment and adds two environment variables for the Elasticsearch username and password so Kibana can authenticate:

containers:
  - name: kibana
    image: docker.elastic.co/kibana/kibana:7.1.0
    ports:
      - containerPort: 5601
    env:
      - name: ELASTICSEARCH_USERNAME
        value: "elastic_user"
      - name: ELASTICSEARCH_PASSWORD
        value: "elasticPass123"

He applies the change, waits for Kibana to start, and refreshes the UI. This time a login screen appears — the login Kibana never had when it was unauthenticated. Bob signs in as elastic_user, and his profile shows in the corner. To check from the terminal, he calls Kibana’s status API on the node port with the same credentials:

$ curl -u elastic_user:elasticPass123 localhost:30601/api/status

These example files write the password straight into manifests for demonstration so you can see how the credentials flow. In a real cluster, place credentials in a Kubernetes Secret and reference them from the Deployment/StatefulSet; do not commit or store plaintext passwords in version control.

Locking down network access with NetworkPolicy

So Elasticsearch is locked. But is it safe? Bob tests it from an unrelated place. He goes into a web server pod in the default namespace and calls Elasticsearch’s in-cluster address with the username and password:

$ kubectl exec -it web-server-bf5bd9d79-kcffb -n default -- \
  curl -u elastic_user:elasticPass123 \
  http://elasticsearch.elastic-stack.svc.cluster.local:9200

Elasticsearch answers. A password only proves you know the password — it doesn’t stop a pod that has no business here from reaching the cluster. That’s what the guest list (NetworkPolicy) is for. Bob writes a NetworkPolicy manifest, names it efk-stack-policy, and places it in the elastic-stack namespace. The policy restricts ingress and egress so that pods in the Elastic Stack namespace may only accept traffic from — and send traffic to — pods in the same namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: efk-stack-policy
  namespace: elastic-stack
spec:
  podSelector:
    matchLabels: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
    - from:
      - podSelector: {}
        namespaceSelector:
          matchLabels:
            name: elastic-stack
  egress:
    - to:
      - podSelector: {}
        namespaceSelector:
          matchLabels:
            name: elastic-stack

He applies the policy and repeats the same curl from the web server pod.

$ kubectl apply -f networkpolicy-efk.yaml
networkpolicy.networking.k8s.io/efk-stack-policy created

This time the curl just sits and eventually times out — the request never reaches Elasticsearch.

A stylized diagram titled "It Never Reached Elasticsearch" showing a web-server in a "default" namespace on the left blocked from reaching an "elastic-stack" namespace on the right that contains elasticsearch-0, fluentd, and kibana services. A lock icon and an X mark highlight that traffic out to Elasticsearch was prevented.

The pod asked the policy which namespace was on its whitelist, and it returned elastic-stack. Now unauthorized namespaces are blocked.

Fluentd needs credentials and certificate verification

Now there’s a casualty. Picture the line from the app to Elasticsearch, with Fluentd sitting in the middle of it — suddenly stuck. Bob reads the Fluentd pod logs and finds an authorization error:

$ kubectl logs -l app=fluentd
[warn]: #0 failed to flush the buffer.

Elasticsearch wants a password, and Fluentd never had one configured. Kibana shows no new data. Bob opens the Fluentd config, finds the match section that sends log lines to Elasticsearch, and makes two changes:

Flip the certificate check from false to true so Fluentd verifies Elasticsearch’s HTTP/TLS certificate (this assumes Elasticsearch’s HTTP/REST API is served over HTTPS).
Add the Elasticsearch username and password so Fluentd can authenticate.

The updated match section looks like this:

<match **>
  @type elasticsearch
  @id out_es
  @log_level "info"
  include_tag_key true
  host "elasticsearch.elastic-stack.svc.cluster.local"
  port 9200
  path ""
  scheme https
  ssl_verify true
  ssl_version TLSv1_2
  user elastic_user
  password elasticPass123
</match>

Then he deletes the Fluentd pod; the DaemonSet starts a fresh pod with the new config. This time the logs show batches of lines being pushed, and Kibana fills with data again. A quick grep for the certificate option confirms it’s now true:

$ grep -n "ssl_verify" /etc/fluentd/fluent.conf
12:  ssl_verify true

So Bob’s logs are safe. He searches for USER1 in Kibana, and a single log entry comes back with a timestamp and a message:

USER1
02:14 USER1 logged in

That’s all. But which pod wrote it? Which environment? The client can see those fields in the log document. Behind it, the stack keeps creating one new Fluentd index every day, and none have been deleted. How does one make each line carry the context he needs?

Next steps (quick checklist)

Area	Why it matters	Action example
Elasticsearch security	Prevents unauthenticated access to the data store	Enable X-Pack, TLS for transport and HTTP, manage users
Secrets handling	Avoid plaintext secrets in manifests	Store credentials in Kubernetes `Secret` and reference them
Network segmentation	Limits which namespaces/pods can reach Elastic components	Use `NetworkPolicy` to restrict ingress/egress per namespace
Fluentd authentication & TLS	Ensures logs are delivered securely and verified	Configure `ssl_verify true` and provide `user` + `password`
Observability	Ensure logs include pod and environment context	Enrich Fluentd records with pod metadata and labels

References and further reading:

Introduction to EFK on Kubernetes

Deploying Elasticsearch

Collecting Logs With Fluentd

Visualizing Logs With Kibana

Monitoring and Alerting

Scaling the EFK Stack

Securing the EFK Stack

Advanced EFK Configuration

Enable X-Pack security and TLS for Elasticsearch

Create a superuser inside the Elasticsearch pod

Kibana must authenticate to Elasticsearch

Locking down network access with NetworkPolicy

Fluentd needs credentials and certificate verification

Next steps (quick checklist)

Watch Video

Practice Lab

​Enable X-Pack security and TLS for Elasticsearch

​Create a superuser inside the Elasticsearch pod

​Kibana must authenticate to Elasticsearch

​Locking down network access with NetworkPolicy

​Fluentd needs credentials and certificate verification

​Next steps (quick checklist)

Watch Video

Practice Lab

Enable X-Pack security and TLS for Elasticsearch

Create a superuser inside the Elasticsearch pod

Kibana must authenticate to Elasticsearch

Locking down network access with NetworkPolicy

Fluentd needs credentials and certificate verification

Next steps (quick checklist)