Scaling Problem and Solution

What has grown too small? Bob opens Grafana and sees the symptoms immediately. Three months ago the app emitted only a few log lines per minute. Now multiple apps share the cluster and logs pour in — but the EFK stack is still the same size it was on day one. From the team’s perspective that looks like slow searches and delayed logs. A search for USER1 in Discover takes a long time to return, and the newest lines are minutes behind real time:

USER1
08:41 USER1 logged in
08:42 USER1 opened page

On the busiest node a single Fluentd pod is reading a file that grows faster than it can forward, while the lone Elasticsearch pod (Elasticsearch 0) has CPU pinned near the top of the chart all day. Bob has two cardinal options: make a pod bigger (vertical scaling) or add more pods (horizontal scaling). Below we explain each approach, how Fluentd buffers behave under load, how Elasticsearch can be sharded, and how Kubernetes autoscaling can help.

Vertical scaling: make a pod bigger

Kubernetes allocates CPU in millicores (m). 1000m = 1 CPU core. In this example the Elasticsearch container is configured with requests.cpu: "250m" and limits.cpu: "500m" which guarantees a quarter of a core and allows up to half a core. Example container resource settings (Deployment / StatefulSet spec):

# example: container resources for Elasticsearch
resources:
  requests:
    cpu: "250m"
    memory: "1Gi"
  limits:
    cpu: "500m"
    memory: "2Gi"

This is called vertical scaling. It’s quick to apply but limited by the node size — one pod cannot be bigger than the node where it runs.

Horizontal scaling: add more pods

Start with Fluentd. The common pattern uses a DaemonSet to ensure one Fluentd pod runs on every node; a newly added node automatically gets its own collector.

An infographic titled "One Fluentd per Node" showing three server-like nodes labeled controlplane, node01 (both with a small Fluentd mascot) and an empty node02. The word "DaemonSet" appears below the nodes.

A DaemonSet is great for even coverage, but it doesn’t help when one node is much noisier than the rest. That busy node needs extra collectors. You can add them with a ReplicaSet or Deployment to ensure a fixed number of identical Fluentd pods. For example, running two additional Fluentd replicas on the busy node gives three collectors there (1 from DaemonSet + 2 replicas). To schedule extra replicas on a specific node you typically use node selectors, node affinity, or label that node and target it with a deployment spec.

Fluentd buffer behavior

When Elasticsearch becomes slow, Fluentd does not simply drop logs. It buffers them and sends in batches called chunks. You can tune how often buffers are flushed, the maximum chunk size, and retry behavior (including retry forever).

A stylized diagram titled "Tuning the Buffer" with a central buffer block labeled "2M" flanked by two glowing, robot-like devices on platforms — a smiling one on the left and a sleepy one on the right.

Example Fluentd buffer configuration:

<match **>
  @type elasticsearch
  [...]
  <buffer tag,time>
    @type memory
    flush_interval 5s
    chunk_limit_size 2m
    retry_forever true
  </buffer>
</match>

Tuning these values (for example, flush_interval, chunk_limit_size, and retry strategy) reduces tailing and prevents loss when Elasticsearch is under transient load.

Sharding Elasticsearch

Can Bob simply add more Elasticsearch pods? Only if the data can be split across nodes.

Elasticsearch divides an index into shards. Different Elasticsearch nodes can hold different shards and search shards in parallel — that’s sharding. If an index currently has only one primary shard, adding more Elasticsearch pods won’t distribute that index’s data. Bob’s index has one shard; he wants five. You cannot safely change the number of primary shards for an existing, populated index. The correct approach is:

Create a new index with the desired number of primary shards (and replicas).
Reindex (copy) documents from the old index into the new one.
Point your writes/aliases to the new index once reindexing is complete.

Warning: you cannot safely reshard an index that’s already full; reindexing into a new index is required.

You cannot change the number of primary shards for an existing index after it is created. Create a new index with the desired number_of_shards and reindex the data into it instead.

Example: create a new index with 5 primary shards via the Elasticsearch REST API:

curl -XPUT "http://elasticsearch:9200/logs_v2" -H "Content-Type: application/json" -d '{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'

(The reindex step is an essential follow-up but is not shown here.)

Autoscaling with the HPA

Adding pods by hand is useful, but load often fluctuates: busy during the day, quiet at night. You don’t want to manually add pods at 2 AM and remove them at 6 AM.

Kubernetes can automate this with the Horizontal Pod Autoscaler (HPA). To use the HPA you must expose pod resource metrics to the cluster; install the Metrics Server first.

Install the Kubernetes Metrics Server so the cluster can read pod CPU and memory metrics. The HPA relies on this data to scale pods automatically. See the Kubernetes docs: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server

An infographic titled "The Horizontal Pod Autoscaler" showing cartoon pod icons on a platform with a rule labeled "1–3 pods" and a dotted slot for scaling. To the right is a circular 50% CPU gauge labeled "metrics-server."

Example HPA manifest (CPU-based autoscaling using autoscaling/v1):

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: elasticsearch-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: elasticsearch
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80

Choose HPA thresholds carefully. A poorly chosen targetCPUUtilizationPercentage can cause frequent scale-up/scale-down oscillation.

Quick reference: compare strategies

Strategy	When to use	Example / Notes
Vertical scaling	CPU/memory pressure is steady and predictable on a single pod	Increase `resources.requests`/`limits` in the pod spec
Horizontal scaling (pods)	One node or service receives much more load than others	Add replicas with nodeAffinity or label a busy node and deploy extra collectors
Fluentd buffer tuning	Transient Elasticsearch slowness causes delayed or batched logs	Tune `flush_interval`, `chunk_limit_size` and retries in Fluentd config
Sharding (Elasticsearch)	Index size or query load needs to be distributed across nodes	Create a new index with more `number_of_shards` and reindex
HPA + Metrics Server	Load fluctuates over time; want automatic scaling	Install Metrics Server and configure HPA for CPU/memory-based autoscaling

Summary — when to do what

If pressure is steady and sustained: increase resources (vertical scale) or add permanent pods and ensure indices are sharded appropriately.
If pressure is variable (peaks and troughs): use the HPA with the Metrics Server to autoscale pods.
Fluentd recommendations: use a DaemonSet for one-per-node collection, add targeted replicas on busy nodes, and tune buffers to prevent dropped logs during transient Elasticsearch slowness.
Elasticsearch recommendations: plan your shard strategy before scaling nodes — add nodes only when indices can be spread across shards.

Next steps

Now, open a terminal and apply changes as needed:

Update resource requests/limits for Elasticsearch pods.
Adjust Fluentd buffer settings in its ConfigMap.
Deploy additional Fluentd replicas (with node selectors/affinity for busy nodes).
Create a new, sharded Elasticsearch index and reindex old data into it.
Install the Metrics Server and apply an HPA manifest for Elasticsearch.

Copying (reindexing) old data into a new, sharded index is required before you can benefit from added Elasticsearch nodes; detailed reindex instructions are not included here.

Links and references

Kubernetes Metrics Server: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server
Horizontal Pod Autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Introduction to EFK on Kubernetes

Deploying Elasticsearch

Collecting Logs With Fluentd

Visualizing Logs With Kibana

Monitoring and Alerting

Scaling the EFK Stack

Securing the EFK Stack

Advanced EFK Configuration

Scaling Problem and Solution

Vertical scaling: make a pod bigger

Horizontal scaling: add more pods

Fluentd buffer behavior

Sharding Elasticsearch

Autoscaling with the HPA

Quick reference: compare strategies

Summary — when to do what

Next steps

Links and references

Watch Video

​Vertical scaling: make a pod bigger

​Horizontal scaling: add more pods

​Fluentd buffer behavior

​Sharding Elasticsearch

​Autoscaling with the HPA

​Quick reference: compare strategies

​Summary — when to do what

​Next steps

​Links and references

Watch Video

Vertical scaling: make a pod bigger

Horizontal scaling: add more pods

Fluentd buffer behavior

Sharding Elasticsearch

Autoscaling with the HPA

Quick reference: compare strategies

Summary — when to do what

Next steps

Links and references