Hands On Scaling

But here’s the rule now: Find the part of the stack that’s struggling, give that part more room, and check the number instead of trusting the command.

This page walks through how Bob follows that rule across the EFK stack: identify a bottleneck (indices, collectors, or Elasticsearch), give it room (shards, buffers, CPU), and verify using metrics and controlled tests.

Index shard bottleneck — check the number

Bob starts at Kibana → Index Management and inspects today’s Fluentd index (fluentd-<date>). The UI shows “primaries: 1”. What does that “1” mean? It’s the number of primary shards for that index. A shard is a slice of an index that can live on a separate Elasticsearch node and be searched in parallel. With only one primary shard, all index operations and searches hit the same slice — a single bottleneck. Can you safely change the primary shard count on a live index? No. Increasing the number of primary shards on an existing index is not supported and doing so improperly can corrupt or make data unavailable.

Recommended approach: create a new index with the desired shard count, then reindex data from the old index into the new one. This avoids modifying an active index’s shard layout and keeps data safe. Example (assumes Elasticsearch reachable at http://localhost:9200):

# Create a new index with 5 primary shards
curl -X PUT "http://localhost:9200/myindex" -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'

# Verify settings
curl -s "http://localhost:9200/myindex/_settings" | jq

Then copy documents from the old index to the new one using the Reindex API:

curl -X POST "http://localhost:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d '{
  "source": { "index": "fluentd-<date>" },
  "dest": { "index": "myindex" }
}'

Back in Kibana click “Reload Indices” and you should see myindex with five primaries. Reindexing avoids changing shard allocation for an active index and preserves availability.

Before reindexing in production: take a snapshot of the source index (or the cluster) and validate snapshots on a restore cluster to prevent accidental data loss.

Collector (Fluentd) — buffer and replica options

When applications suddenly emit many more log lines, Fluentd buffers events in memory or files until it can flush them to Elasticsearch. A buffer acts as a waiting room that batches events for efficient delivery and retries on failure. Bob updates the Fluentd match configuration to flush batches every 5 seconds and to retry indefinitely if Elasticsearch becomes unavailable:

<match **>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true

  <buffer>
    @type file
    chunk_limit_size 2M
    flush_interval 5s
    retry_forever true
  </buffer>
</match>

After saving the configuration, Bob deletes the Fluentd pod. The DaemonSet controller notices the missing pod and schedules a new one; the new pod picks up the updated configuration on startup. Note on placement: a DaemonSet runs one pod per node by design. That ensures cluster-wide collection but can become a bottleneck if one node has far more workload than others. To let a busy node run multiple collectors, run the fluentd pod spec as a ReplicaSet (or Deployment) with multiple replicas so the scheduler can place two or more collectors on the same node. Example ReplicaSet manifest:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: fluentd-replicaset
  namespace: elastic-stack
  labels:
    app: fluentd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:latest
        # mount config, volumes, etc. as needed

After applying:

kubectl get rs -n elastic-stack fluentd-replicaset
# NAME                    DESIRED   CURRENT   READY
# fluentd-replicaset      2         2         2

Use DaemonSets when you need exactly one collector per node (e.g., to access node-local logs). Use ReplicaSets/Deployments when you need multiple collectors on a single node to absorb bursty or uneven traffic.

Elasticsearch pod sizing — vertical scaling (requests & limits)

If Elasticsearch pods are hitting CPU or heap alerts, give them more resources. Bob edits the Elasticsearch StatefulSet container resources to request 250m (0.25 core) and limit 500m (0.5 core). Kubernetes will reserve the requested CPU and enforce the hard ceiling defined by the limit. Example StatefulSet container resources snippet:

spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    resources:
      requests:
        cpu: "250m"
      limits:
        cpu: "500m"
    ports:
    - containerPort: 9200
      name: http
    - containerPort: 9300
      name: transport

An infographic titled "Reserved and Ceiling" showing a single CPU core column with a green "Request" at 250m and a red "Limit" at 500m. A small blue mascot sits on a server platform and points toward the CPU usage indicators.

Apply and confirm:

kubectl apply -f es-statefulset.yaml
kubectl get statefulset elasticsearch -n elastic-stack -o yaml | yq '.spec.template.spec.containers[] | {name: .name, resources: .resources}'

Vertical scaling increases capacity per pod. If the cluster needs more overall throughput or high availability, combine vertical sizing with horizontal scaling.

Horizontal scaling — HPA + Metrics Server

For automatic horizontal scaling, use a HorizontalPodAutoscaler (HPA). The HPA needs metrics (typically CPU) from the Kubernetes Metrics Server. Install Metrics Server (example uses v0.5.0 and the lab-only --kubelet-insecure-tls flag):

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml
# Edit components.yaml: add the following arg to the metrics-server container:
# - --kubelet-insecure-tls

Snippet to add inside components.yaml:

containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=443
    - --kubelet-insecure-tls

The --kubelet-insecure-tls option disables kubelet certificate verification. It is fine for a lab environment but should NOT be used in production.

Apply and check:

kubectl apply -f components.yaml
# serviceaccount/metrics-server created
# deployment.apps/metrics-server created
kubectl get pods -n kube-system
# NAME                             READY   STATUS    RESTARTS   AGE
# metrics-server-6dfddc5fb8-x2v7k  1/1     Running   0          40s

Create an HPA targeting the Elasticsearch StatefulSet:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: elasticsearch-autoscaler
  namespace: elastic-stack
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: elasticsearch
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80

This HPA attempts to keep average CPU utilization near 80% across StatefulSet pods and will scale between 1 and 3 replicas. To test HPA behavior, force a temporary CPU spike (e.g., trigger heavy startup CPU use by deleting elasticsearch-0 so a new pod rebuilds and overshoots the target). The HPA observes the increased CPU usage via Metrics Server and scales up:

kubectl delete pod elasticsearch-0 -n elastic-stack

# Observe HPA
kubectl get hpa elasticsearch-autoscaler -n elastic-stack
# NAME                         REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# elasticsearch-autoscaler     StatefulSet/elasticsearch     120%/80%  1         3         2          2m

HPA reacts to metrics — it won’t proactively create pods without observed load.

Quick comparison: scaling options

Approach	When to use	Example / Notes
Increase shards (create new index & reindex)	Index-level search/ingest bottleneck	Create index with `number_of_shards: 5` and use `_reindex` API
Fluentd buffer tuning	Short-term burst handling and retry resilience	Use `chunk_limit_size`, `flush_interval`, `retry_forever` in `<buffer>`
DaemonSet	One collector per node (node-local logs)	Guarantees presence on every node
ReplicaSet / Deployment	Allow multiple collectors on a busy node	Set `replicas: N` so scheduler may place >1 on a node
Vertical scaling (requests/limits)	Pod needs more CPU or memory	`requests: cpu: "250m"`, `limits: cpu: "500m"`
Horizontal scaling (HPA)	Scale out based on observed metrics	Requires Metrics Server; HPA uses CPU or custom metrics

Secure your Elasticsearch

A properly scaled stack is still vulnerable if Elasticsearch is left exposed without authentication, authorization, and TLS. In Bob’s lab a test pod issued a single request to the open Elasticsearch endpoint and could retrieve every log line from the month — potentially exposing usernames and PII.

Secure Elasticsearch immediately in non-lab environments:

Enable authentication and role-based access control.
Use TLS (HTTPS) for both HTTP and transport layers.
Restrict network access with NetworkPolicies and service-level controls.
Audit and monitor access to sensitive indices.

Links and references

Kubernetes Metrics Server: https://github.com/kubernetes-sigs/metrics-server
Kubernetes Horizontal Pod Autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Elasticsearch Reindex API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
Kibana Index Management: https://www.elastic.co/guide/en/kibana/current/index-management.html
Kubernetes docs (resources/requests/limits): https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Introduction to EFK on Kubernetes

Deploying Elasticsearch

Collecting Logs With Fluentd

Visualizing Logs With Kibana

Monitoring and Alerting

Scaling the EFK Stack

Securing the EFK Stack

Advanced EFK Configuration

Index shard bottleneck — check the number

Collector (Fluentd) — buffer and replica options

Elasticsearch pod sizing — vertical scaling (requests & limits)

Horizontal scaling — HPA + Metrics Server

Quick comparison: scaling options

Secure your Elasticsearch

Links and references

Watch Video

Practice Lab

​Index shard bottleneck — check the number

​Collector (Fluentd) — buffer and replica options

​Elasticsearch pod sizing — vertical scaling (requests & limits)

​Horizontal scaling — HPA + Metrics Server

​Quick comparison: scaling options

​Secure your Elasticsearch

​Links and references

Watch Video

Practice Lab

Index shard bottleneck — check the number

Collector (Fluentd) — buffer and replica options

Elasticsearch pod sizing — vertical scaling (requests & limits)

Horizontal scaling — HPA + Metrics Server

Quick comparison: scaling options

Secure your Elasticsearch

Links and references