> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling Problem and Solution

> Guidance for scaling and tuning an EFK logging stack on Kubernetes covering pod sizing, Fluentd buffering, Elasticsearch sharding, and autoscaling with Metrics Server and HPA

What has grown too small?

Bob opens Grafana and sees the symptoms immediately.

Three months ago the app emitted only a few log lines per minute. Now multiple apps share the cluster and logs pour in — but the EFK stack is still the same size it was on day one.

From the team's perspective that looks like slow searches and delayed logs. A search for `USER1` in Discover takes a long time to return, and the newest lines are minutes behind real time:

```text theme={null}
USER1
08:41 USER1 logged in
08:42 USER1 opened page
```

On the busiest node a single Fluentd pod is reading a file that grows faster than it can forward, while the lone Elasticsearch pod (Elasticsearch 0) has CPU pinned near the top of the chart all day.

Bob has two cardinal options: make a pod bigger (vertical scaling) or add more pods (horizontal scaling). Below we explain each approach, how Fluentd buffers behave under load, how Elasticsearch can be sharded, and how Kubernetes autoscaling can help.

## Vertical scaling: make a pod bigger

Kubernetes allocates CPU in millicores (m). 1000m = 1 CPU core. In this example the Elasticsearch container is configured with `requests.cpu: "250m"` and `limits.cpu: "500m"` which guarantees a quarter of a core and allows up to half a core.

Example container resource settings (Deployment / StatefulSet spec):

```yaml theme={null}
# example: container resources for Elasticsearch
resources:
  requests:
    cpu: "250m"
    memory: "1Gi"
  limits:
    cpu: "500m"
    memory: "2Gi"
```

This is called vertical scaling. It's quick to apply but limited by the node size — one pod cannot be bigger than the node where it runs.

## Horizontal scaling: add more pods

Start with Fluentd. The common pattern uses a DaemonSet to ensure one Fluentd pod runs on every node; a newly added node automatically gets its own collector.

![An infographic titled "One Fluentd per Node" showing three server-like nodes labeled controlplane, node01 (both with a small Fluentd mascot) and an empty node02. The word "DaemonSet" appears below the nodes.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Scaling-Problem-and-Solution/one-fluentd-per-node-daemonset.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=81c725f9bb802888b2eaa7bbd3986af1)

A DaemonSet is great for even coverage, but it doesn't help when one node is much noisier than the rest. That busy node needs extra collectors. You can add them with a ReplicaSet or Deployment to ensure a fixed number of identical Fluentd pods. For example, running two additional Fluentd replicas on the busy node gives three collectors there (1 from DaemonSet + 2 replicas).

To schedule extra replicas on a specific node you typically use node selectors, node affinity, or label that node and target it with a deployment spec.

### Fluentd buffer behavior

When Elasticsearch becomes slow, Fluentd does not simply drop logs. It buffers them and sends in batches called chunks. You can tune how often buffers are flushed, the maximum chunk size, and retry behavior (including retry forever).

![A stylized diagram titled "Tuning the Buffer" with a central buffer block labeled "2M" flanked by two glowing, robot-like devices on platforms — a smiling one on the left and a sleepy one on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Scaling-Problem-and-Solution/tuning-buffer-2m-robots-smiling-sleepy.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=16a4cf2736b548caafc9aec0c2e87704)

Example Fluentd buffer configuration:

```xml theme={null}
<match **>
  @type elasticsearch
  [...]
  <buffer tag,time>
    @type memory
    flush_interval 5s
    chunk_limit_size 2m
    retry_forever true
  </buffer>
</match>
```

Tuning these values (for example, `flush_interval`, `chunk_limit_size`, and retry strategy) reduces tailing and prevents loss when Elasticsearch is under transient load.

## Sharding Elasticsearch

Can Bob simply add more Elasticsearch pods? Only if the data can be split across nodes.

![A dark teal slide titled "Can the Data Be Split" showing a teal progress bar labeled "fluentd index" and a small blue robot character sitting on a platform. The KodeKloud logo appears in the bottom-left.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Scaling-Problem-and-Solution/fluentd-index-data-split-robot.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=26e02c4170374771b030c1a20e78eed3)

Elasticsearch divides an index into shards. Different Elasticsearch nodes can hold different shards and search shards in parallel — that’s sharding. If an index currently has only one primary shard, adding more Elasticsearch pods won't distribute that index's data.

Bob’s index has one shard; he wants five. You cannot safely change the number of primary shards for an existing, populated index. The correct approach is:

1. Create a new index with the desired number of primary shards (and replicas).
2. Reindex (copy) documents from the old index into the new one.
3. Point your writes/aliases to the new index once reindexing is complete.

Warning: you cannot safely reshard an index that's already full; reindexing into a new index is required.

You cannot change the number of primary shards for an existing index after it is created. Create a new index with the desired `number_of_shards` and reindex the data into it instead.

Example: create a new index with 5 primary shards via the Elasticsearch REST API:

```bash theme={null}
curl -XPUT "http://elasticsearch:9200/logs_v2" -H "Content-Type: application/json" -d '{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'
```

(The reindex step is an essential follow-up but is not shown here.)

## Autoscaling with the HPA

Adding pods by hand is useful, but load often fluctuates: busy during the day, quiet at night. You don't want to manually add pods at 2 AM and remove them at 6 AM.

![A dark-themed dashboard titled "The Load Comes and Goes" showing a 24-hour line chart of load with a midday peak and a sad avatar labeled "Bob" at the right. Below the chart are two small blue robot icons sitting on a platform.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Scaling-Problem-and-Solution/dashboard-load-peak-sad-bob-robots.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=527938195f1e943e89e1eb898d619946)

Kubernetes can automate this with the Horizontal Pod Autoscaler (HPA). To use the HPA you must expose pod resource metrics to the cluster; install the Metrics Server first.

Install the Kubernetes Metrics Server so the cluster can read pod CPU and memory metrics. The HPA relies on this data to scale pods automatically. See the Kubernetes docs: [https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server)

![An infographic titled "The Horizontal Pod Autoscaler" showing cartoon pod icons on a platform with a rule labeled "1–3 pods" and a dotted slot for scaling. To the right is a circular 50% CPU gauge labeled "metrics-server."](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Scaling-the-EFK-Stack/Scaling-Problem-and-Solution/horizontal-pod-autoscaler-pods-1to3-metricsserver.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=e0dfce7f6b833ea095a2cc36b45cad99)

Example HPA manifest (CPU-based autoscaling using autoscaling/v1):

```yaml theme={null}
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: elasticsearch-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: elasticsearch
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80
```

Choose HPA thresholds carefully. A poorly chosen `targetCPUUtilizationPercentage` can cause frequent scale-up/scale-down oscillation.

## Quick reference: compare strategies

| Strategy | When to use | Example / Notes |
| - | - | - |
| Vertical scaling | CPU/memory pressure is steady and predictable on a single pod | Increase `resources.requests`/`limits` in the pod spec |
| Horizontal scaling (pods) | One node or service receives much more load than others | Add replicas with nodeAffinity or label a busy node and deploy extra collectors |
| Fluentd buffer tuning | Transient Elasticsearch slowness causes delayed or batched logs | Tune `flush_interval`, `chunk_limit_size` and retries in Fluentd config |
| Sharding (Elasticsearch) | Index size or query load needs to be distributed across nodes | Create a new index with more `number_of_shards` and reindex |
| HPA + Metrics Server | Load fluctuates over time; want automatic scaling | Install Metrics Server and configure HPA for CPU/memory-based autoscaling |

## Summary — when to do what

* If pressure is steady and sustained: increase resources (vertical scale) or add permanent pods and ensure indices are sharded appropriately.
* If pressure is variable (peaks and troughs): use the HPA with the Metrics Server to autoscale pods.
* Fluentd recommendations: use a DaemonSet for one-per-node collection, add targeted replicas on busy nodes, and tune buffers to prevent dropped logs during transient Elasticsearch slowness.
* Elasticsearch recommendations: plan your shard strategy before scaling nodes — add nodes only when indices can be spread across shards.

## Next steps

Now, open a terminal and apply changes as needed:

* Update resource `requests`/`limits` for Elasticsearch pods.
* Adjust Fluentd buffer settings in its ConfigMap.
* Deploy additional Fluentd replicas (with node selectors/affinity for busy nodes).
* Create a new, sharded Elasticsearch index and reindex old data into it.
* Install the Metrics Server and apply an HPA manifest for Elasticsearch.

Copying (reindexing) old data into a new, sharded index is required before you can benefit from added Elasticsearch nodes; detailed reindex instructions are not included here.

## Links and references

* Kubernetes Metrics Server: [https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server)
* Horizontal Pod Autoscaler: [https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
* Elasticsearch documentation: [https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)

> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/30cd6e2d-849b-4091-b20c-f3d4d4b5a640/lesson/ccd3b391-f58b-44f4-a454-a93a93ee51ab)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.