> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hands On Scaling

> How to identify and fix scaling bottlenecks in the EFK stack by resizing shards, tuning Fluentd buffers and replicas, adjusting Elasticsearch resources, using HPA, and validating with metrics

But here's the rule now: Find the part of the stack that's struggling, give that part more room, and check the number instead of trusting the command.

![A slide titled "Bob's Rule" showing a three-step workflow—Find the part, Give it room, Check the number—represented by icons and arrows. The slide has a dark background with a small KodeKloud logo in the bottom-left.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Hands-On-Scaling/bobs-rule-three-step-workflow.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=0915b461064d08eb3d036aa519491b20)

This page walks through how Bob follows that rule across the EFK stack: identify a bottleneck (indices, collectors, or Elasticsearch), give it room (shards, buffers, CPU), and verify using metrics and controlled tests.

## Index shard bottleneck — check the number

Bob starts at Kibana → Index Management and inspects today's Fluentd index (`fluentd-<date>`). The UI shows "primaries: 1".

What does that "1" mean? It's the number of primary shards for that index. A shard is a slice of an index that can live on a separate Elasticsearch node and be searched in parallel. With only one primary shard, all index operations and searches hit the same slice — a single bottleneck.

Can you safely change the primary shard count on a live index? No. Increasing the number of primary shards on an existing index is not supported and doing so improperly can corrupt or make data unavailable.

![A dark-themed dashboard screen that reads "Not on a Live Index," showing a circular widget labeled "1 shards" and a disabled input box with an X. A small blue mascot sits on a platform and the KodeKloud logo appears in the bottom-left.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Hands-On-Scaling/not-on-live-index-kodekloud.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=29f363328c504da76527c870a4e04256)

Recommended approach: create a new index with the desired shard count, then reindex data from the old index into the new one. This avoids modifying an active index's shard layout and keeps data safe.

Example (assumes Elasticsearch reachable at `http://localhost:9200`):

```bash theme={null}
# Create a new index with 5 primary shards
curl -X PUT "http://localhost:9200/myindex" -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'

# Verify settings
curl -s "http://localhost:9200/myindex/_settings" | jq
```

Then copy documents from the old index to the new one using the Reindex API:

```bash theme={null}
curl -X POST "http://localhost:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d '{
  "source": { "index": "fluentd-<date>" },
  "dest": { "index": "myindex" }
}'
```

Back in Kibana click "Reload Indices" and you should see `myindex` with five primaries. Reindexing avoids changing shard allocation for an active index and preserves availability.

Before reindexing in production: take a snapshot of the source index (or the cluster) and validate snapshots on a restore cluster to prevent accidental data loss.

## Collector (Fluentd) — buffer and replica options

When applications suddenly emit many more log lines, Fluentd buffers events in memory or files until it can flush them to Elasticsearch. A buffer acts as a waiting room that batches events for efficient delivery and retries on failure.

Bob updates the Fluentd match configuration to flush batches every 5 seconds and to retry indefinitely if Elasticsearch becomes unavailable:

```xml theme={null}
<match **>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true

  <buffer>
    @type file
    chunk_limit_size 2M
    flush_interval 5s
    retry_forever true
  </buffer>
</match>
```

After saving the configuration, Bob deletes the Fluentd pod. The DaemonSet controller notices the missing pod and schedules a new one; the new pod picks up the updated configuration on startup.

Note on placement: a DaemonSet runs one pod per node by design. That ensures cluster-wide collection but can become a bottleneck if one node has far more workload than others. To let a busy node run multiple collectors, run the fluentd pod spec as a ReplicaSet (or Deployment) with multiple replicas so the scheduler can place two or more collectors on the same node.

Example ReplicaSet manifest:

```yaml theme={null}
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: fluentd-replicaset
  namespace: elastic-stack
  labels:
    app: fluentd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:latest
        # mount config, volumes, etc. as needed
```

After applying:

```bash theme={null}
kubectl get rs -n elastic-stack fluentd-replicaset
# NAME                    DESIRED   CURRENT   READY
# fluentd-replicaset      2         2         2
```

Use DaemonSets when you need exactly one collector per node (e.g., to access node-local logs). Use ReplicaSets/Deployments when you need multiple collectors on a single node to absorb bursty or uneven traffic.

## Elasticsearch pod sizing — vertical scaling (requests & limits)

If Elasticsearch pods are hitting CPU or heap alerts, give them more resources. Bob edits the Elasticsearch StatefulSet container resources to request 250m (0.25 core) and limit 500m (0.5 core). Kubernetes will reserve the requested CPU and enforce the hard ceiling defined by the limit.

Example StatefulSet container resources snippet:

```yaml theme={null}
spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    resources:
      requests:
        cpu: "250m"
      limits:
        cpu: "500m"
    ports:
    - containerPort: 9200
      name: http
    - containerPort: 9300
      name: transport
```

![An infographic titled "Reserved and Ceiling" showing a single CPU core column with a green "Request" at 250m and a red "Limit" at 500m. A small blue mascot sits on a server platform and points toward the CPU usage indicators.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Hands-On-Scaling/reserved-ceiling-cpu-250m-500m.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=598242ece4283a5f8a8dcd97bf7bee00)

Apply and confirm:

```bash theme={null}
kubectl apply -f es-statefulset.yaml
kubectl get statefulset elasticsearch -n elastic-stack -o yaml | yq '.spec.template.spec.containers[] | {name: .name, resources: .resources}'
```

Vertical scaling increases capacity per pod. If the cluster needs more overall throughput or high availability, combine vertical sizing with horizontal scaling.

## Horizontal scaling — HPA + Metrics Server

For automatic horizontal scaling, use a HorizontalPodAutoscaler (HPA). The HPA needs metrics (typically CPU) from the Kubernetes Metrics Server.

Install Metrics Server (example uses v0.5.0 and the lab-only `--kubelet-insecure-tls` flag):

```bash theme={null}
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml
# Edit components.yaml: add the following arg to the metrics-server container:
# - --kubelet-insecure-tls
```

Snippet to add inside `components.yaml`:

```yaml theme={null}
containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=443
    - --kubelet-insecure-tls
```

The `--kubelet-insecure-tls` option disables kubelet certificate verification. It is fine for a lab environment but should NOT be used in production.

Apply and check:

```bash theme={null}
kubectl apply -f components.yaml
# serviceaccount/metrics-server created
# deployment.apps/metrics-server created
kubectl get pods -n kube-system
# NAME                             READY   STATUS    RESTARTS   AGE
# metrics-server-6dfddc5fb8-x2v7k  1/1     Running   0          40s
```

Create an HPA targeting the Elasticsearch StatefulSet:

```yaml theme={null}
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: elasticsearch-autoscaler
  namespace: elastic-stack
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: elasticsearch
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80
```

This HPA attempts to keep average CPU utilization near 80% across StatefulSet pods and will scale between 1 and 3 replicas.

To test HPA behavior, force a temporary CPU spike (e.g., trigger heavy startup CPU use by deleting `elasticsearch-0` so a new pod rebuilds and overshoots the target). The HPA observes the increased CPU usage via Metrics Server and scales up:

```bash theme={null}
kubectl delete pod elasticsearch-0 -n elastic-stack

# Observe HPA
kubectl get hpa elasticsearch-autoscaler -n elastic-stack
# NAME                         REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# elasticsearch-autoscaler     StatefulSet/elasticsearch     120%/80%  1         3         2          2m
```

HPA reacts to metrics — it won't proactively create pods without observed load.

## Quick comparison: scaling options

| Approach | When to use | Example / Notes |
| - | -: | - |
| Increase shards (create new index & reindex) | Index-level search/ingest bottleneck | Create index with `number_of_shards: 5` and use `_reindex` API |
| Fluentd buffer tuning | Short-term burst handling and retry resilience | Use `chunk_limit_size`, `flush_interval`, `retry_forever` in `<buffer>` |
| DaemonSet | One collector per node (node-local logs) | Guarantees presence on every node |
| ReplicaSet / Deployment | Allow multiple collectors on a busy node | Set `replicas: N` so scheduler may place >1 on a node |
| Vertical scaling (requests/limits) | Pod needs more CPU or memory | `requests: cpu: "250m"`, `limits: cpu: "500m"` |
| Horizontal scaling (HPA) | Scale out based on observed metrics | Requires Metrics Server; HPA uses CPU or custom metrics |

## Secure your Elasticsearch

A properly scaled stack is still vulnerable if Elasticsearch is left exposed without authentication, authorization, and TLS. In Bob's lab a test pod issued a single request to the open Elasticsearch endpoint and could retrieve every log line from the month — potentially exposing usernames and PII.

![A dark-themed illustration titled "Open to Anyone" with a central list showing "USER1 logged in" through "USER5 logged in." Two cute server avatars sit on platforms at left and right (the left one labeled "test pod") and a KodeKloud logo appears in the corner.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Hands-On-Scaling/open-to-anyone-user-list-kodekloud.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=3485fbd82060b5777a9981d9714e88e9)

Secure Elasticsearch immediately in non-lab environments:

* Enable authentication and role-based access control.
* Use TLS (HTTPS) for both HTTP and transport layers.
* Restrict network access with NetworkPolicies and service-level controls.
* Audit and monitor access to sensitive indices.

## Links and references

* Kubernetes Metrics Server: [https://github.com/kubernetes-sigs/metrics-server](https://github.com/kubernetes-sigs/metrics-server)
* Kubernetes Horizontal Pod Autoscaler: [https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
* Elasticsearch Reindex API: [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)
* Kibana Index Management: [https://www.elastic.co/guide/en/kibana/current/index-management.html](https://www.elastic.co/guide/en/kibana/current/index-management.html)
* Kubernetes docs (resources/requests/limits): [https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)

> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/30cd6e2d-849b-4091-b20c-f3d4d4b5a640/lesson/86a85403-6496-4156-a191-3eb5f6a01c2e)

> [card] [Practice Lab](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/30cd6e2d-849b-4091-b20c-f3d4d4b5a640/lesson/884aa849-fc2d-4721-94a5-b3bf60eda58c)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.