Skip to main content
The pods are running, but are logs actually arriving at the search backend? A running pod only proves the container started — it doesn’t prove that any log lines completed the pipeline. After you build a logging pipeline, always verify delivery at the far end. In this setup the far end is Elasticsearch; Bob can reach the Elasticsearch service on NodePort 30200.
A simple network diagram titled "Look at the Far End" showing a user labeled "Bob" sending traffic to a Service (NodePort 30200) that forwards to an elasticsearch-0 pod running on a controlplane node. The graphic illustrates the request path from client to backend.

​
Query Elasticsearch to confirm documents arrive

Start by asking Elasticsearch for everything. The _search endpoint with q=*:* and pretty returns all documents in a readable JSON format:
First try — you may see no results. Fluentd buffers and flushes periodically (this configuration flushes every 5 seconds), and the index must be created the first time data arrives. An initial response can look like:
Fluentd buffers records and flushes them on a schedule (e.g., every 5 seconds). If the first search returns no hits, wait a few seconds and try again.
On the next request you should start seeing documents in hits. Each document represents a log line parsed into fields by Fluentd. Example document returned from Elasticsearch:
This shows:
  • A log field containing the application message.
  • A stream field indicating stdout.
  • An index named with the fluentd prefix plus the date (as configured).
  • A kubernetes.pod_name field tying the document back to the app pod.

​
Verify the logs match the pod output

To prove end-to-end delivery, compare the pod’s logs with the documents in Elasticsearch. Bob places two panes side-by-side: one running kubectl logs and the other showing the Elasticsearch JSON results. Example pod output:
Corresponding Elasticsearch hits (trimmed):
Same lines, two places: one copy still on the node and another stored in Elasticsearch. This provides persistence even if the app pod is replaced.

​
Searching for specific values

You can query Elasticsearch directly from the terminal. To search for documents mentioning USER1:
Be aware Elasticsearch returns a default number of documents per search response (default size is 10). A short result doesn’t necessarily mean missing data — increase the size parameter if you expect more results.
Elasticsearch defaults size to 10. If you expect more results, include a larger size value (for example, size=10000) in the query:
After increasing size, additional log lines appear in the results:

​
Quick reference: common terminal checks

​
Summary: what a successful pipeline looks like

End-to-end success means:
  1. The application writes a line to stdout or to a file on the node.
  2. Fluentd tails, parses, buffers, and ships the record.
  3. Elasticsearch receives and stores the document in an index named with the fluentd prefix + date.
  4. You can query Elasticsearch and verify the stored documents match the original pod logs.

​
Next step: make logs accessible to your team

Raw JSON from curl is fine for troubleshooting but not ideal for collaborative investigations. Put a search and visualization UI in front of Elasticsearch — for example, use Kibana or another Elasticsearch-compatible UI — so multiple engineers can run queries, create dashboards, and analyze incidents without parsing raw JSON by hand.

Watch Video

Practice Lab