> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Fluentd Log Forwarding

> How to verify Fluentd forwards Kubernetes pod logs to Elasticsearch by querying indices, comparing pod stdout with ES documents, and using tools like curl and Kibana for visualization

The pods are running, but are logs actually arriving at the search backend?

A running pod only proves the container started — it doesn't prove that any log lines completed the pipeline. After you build a logging pipeline, always verify delivery at the far end. In this setup the far end is Elasticsearch; Bob can reach the Elasticsearch service on NodePort `30200`.

![A simple network diagram titled "Look at the Far End" showing a user labeled "Bob" sending traffic to a Service (NodePort 30200) that forwards to an elasticsearch-0 pod running on a controlplane node. The graphic illustrates the request path from client to backend.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Collecting-Logs-With-Fluentd/Test-Fluentd-Log-Forwarding/bob-nodeport-30200-to-elasticsearch-pod.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=0ffc05ccd19bc3363142505579481712)

## Query Elasticsearch to confirm documents arrive

Start by asking Elasticsearch for everything. The `_search` endpoint with `q=*:*` and `pretty` returns all documents in a readable JSON format:

```bash theme={null}
curl 'http://localhost:30200/_search?q=*:*&pretty'
```

First try — you may see no results. Fluentd buffers and flushes periodically (this configuration flushes every 5 seconds), and the index must be created the first time data arrives. An initial response can look like:

```json theme={null}
{
  "took": 3,
  "timed_out": false,
  "hits": []
}
```

Fluentd buffers records and flushes them on a schedule (e.g., every 5 seconds). If the first search returns no hits, wait a few seconds and try again.

On the next request you should start seeing documents in `hits`. Each document represents a log line parsed into fields by Fluentd. Example document returned from Elasticsearch:

```json theme={null}
{
  "_index": "fluentd-2026.10.13",
  "_type": "_doc",
  "_source": {
    "stream": "stdout",
    "log": "USER1 logged in",
    "kubernetes": {
      "pod_name": "app"
    },
    "time": "08:12:04"
  }
}
```

This shows:

* A `log` field containing the application message.
* A `stream` field indicating `stdout`.
* An index named with the `fluentd` prefix plus the date (as configured).
* A `kubernetes.pod_name` field tying the document back to the `app` pod.

## Verify the logs match the pod output

To prove end-to-end delivery, compare the pod’s logs with the documents in Elasticsearch. Bob places two panes side-by-side: one running `kubectl logs` and the other showing the Elasticsearch JSON results.

Example pod output:

```bash theme={null}
kubectl logs app
08:12:01  USER2 opened page
08:12:04  USER1 logged in
08:12:09  USER3 logged out
08:12:15  USER4 logged in
```

Corresponding Elasticsearch `hits` (trimmed):

```json theme={null}
{
  "hits": [
    {
      "_source": {
        "time": "08:12:01",
        "log": "USER2 opened page"
      }
    },
    {
      "_source": {
        "time": "08:12:04",
        "log": "USER1 logged in"
      }
    },
    {
      "_source": {
        "time": "08:12:09",
        "log": "USER3 logged out"
      }
    },
    {
      "_source": {
        "time": "08:12:15",
        "log": "USER4 logged in"
      }
    }
  ]
}
```

Same lines, two places: one copy still on the node and another stored in Elasticsearch. This provides persistence even if the app pod is replaced.

## Searching for specific values

You can query Elasticsearch directly from the terminal. To search for documents mentioning `USER1`:

```bash theme={null}
curl 'http://localhost:30200/_search?q=USER1&pretty'
```

Be aware Elasticsearch returns a default number of documents per search response (default `size` is 10). A short result doesn't necessarily mean missing data — increase the `size` parameter if you expect more results.

Elasticsearch defaults `size` to 10. If you expect more results, include a larger `size` value (for example, `size=10000`) in the query:

```bash theme={null}
curl 'http://localhost:30200/_search?q=*:*&size=10000&pretty'
```

After increasing `size`, additional log lines appear in the results:

```json theme={null}
{
  "hits": [
    { "_source": { "log": "USER1 logged in" } },
    { "_source": { "log": "USER2 logged out" } },
    { "_source": { "log": "USER3 opened page" } }
    // ...
  ]
}
```

## Quick reference: common terminal checks

| Check | Purpose | Example |
| -: | - | - |
| Basic search | Return all documents (readable JSON) | `curl 'http://localhost:30200/_search?q=*:*&pretty'` |
| Match a term | Find documents mentioning a string | `curl 'http://localhost:30200/_search?q=USER1&pretty'` |
| Return many results | Increase returned results | `curl 'http://localhost:30200/_search?q=*:*&size=10000&pretty'` |
| Verify pod output | Show pod stdout lines to compare with ES | `kubectl logs app` |

## Summary: what a successful pipeline looks like

End-to-end success means:

1. The application writes a line to stdout or to a file on the node.
2. Fluentd tails, parses, buffers, and ships the record.
3. Elasticsearch receives and stores the document in an index named with the `fluentd` prefix + date.
4. You can query Elasticsearch and verify the stored documents match the original pod logs.

## Next step: make logs accessible to your team

Raw JSON from curl is fine for troubleshooting but not ideal for collaborative investigations. Put a search and visualization UI in front of Elasticsearch — for example, use Kibana or another Elasticsearch-compatible UI — so multiple engineers can run queries, create dashboards, and analyze incidents without parsing raw JSON by hand.

## Links and references

* [Fluentd](https://www.fluentd.org)
* [Elasticsearch](https://www.elastic.co/elasticsearch)
* [Kibana](https://www.elastic.co/kibana)

> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/b47732d3-af85-4c4a-8316-6c6ae141d927/lesson/8720ab47-6bcd-4a19-bc11-a03b01a265d2)

> [card] [Practice Lab](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/b47732d3-af85-4c4a-8316-6c6ae141d927/lesson/0e3778c0-1253-4b3c-871b-46a10372f2f7)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.