Skip to main content
Kubernetes operator/controller logs are not a verbatim transcript of every internal action. Instead, they are the flight recorder for your controller: concise clues that help you answer targeted questions about incidents in the cluster. By focusing on identity, motion, and intent vs. reality, you can go from noisy streams to a precise narrative of what happened. Example controller log fragments:
Ask: what did the controller notice? what did it try? why did it come back? Like a flight recorder, you only dig deep when the system shows anomalous behavior. A typical reconcile sequence looks like:
Most lines are routine. Treat the stream as investigative evidence rather than prose — start with a question and use logs to answer it:
  • Which object (namespace/name) is this about?
  • Did the controller enter reconcile?
  • Did it create, update, or skip a child resource?
  • Did the API server accept any change?
A useful log line helps answer at least one of those questions:
Everything else is often background noise. Look 1 — Identity The first thing to locate is identity. In a busy controller, reconciles overlap and multiple objects appear in the same stream. The object name and namespace (e.g., default/demo or webapp demo/blog) are the case number to follow: keep each object’s timeline separate from others.
The image displays a segment titled "Look 1: Identity" explaining the use of namespaces and case numbers to differentiate stories, with folders for "default/demo" and "demo/blog" noting actions like "noticed" and "created svc."
Look 2 — Motion Next, observe motion: a healthy reconcile is a short arc — notice the object, check desired children, make a change if needed, then go quiet. If the same object keeps reappearing, the controller might be “chasing its own shadow”. Watch for log lines that indicate actions which themselves generate more events: requeues, timed retries, or outgoing API calls. These entries often explain subsequent activity in the stream.
The image is a diagram illustrating a process of "Notice" → "Check" → "Change" → "Quiet," with the text "A healthy reconcile is a small arc" under a section titled "Look 2: Motion."
Look 3 — Intent vs. Reality (generation vs. resourceVersion) A key signal is the gap between the user’s intent and the cluster’s observed state. In Kubernetes:
  • generation increments when the user changes the resource spec (intent).
  • resourceVersion changes whenever the object is written/updated in etcd (observed state).
The image compares "generation" and "resourceVersion," indicating that the user's intent is unchanged while the cluster's record keeps changing.
If generation is static while resourceVersion keeps increasing, something in the cluster is modifying the object without a user spec change. Often that “something” is your controller. That pattern helps you detect controller-driven churn or other controllers/webhooks mutating the resource. Log levels — lights and flashlights Think of log levels as lighting:
  • INFO: overhead lighting — useful for regular operations.
  • DEBUG: a flashlight — narrow, detailed illumination you enable for a targeted investigation.
Turn on DEBUG only when you need to inspect internals; otherwise it adds noise and hides the important signals.
Use INFO for day-to-day monitoring. Enable DEBUG temporarily when investigating a specific reconcile or repeated failures. Remember to disable DEBUG after the incident to avoid log overload.
Respect Errors Errors demand attention. When non-actionable events are logged at ERROR, real faults can get buried. Differentiate between benign missing-children (controller will create them) and genuine rejections (API refuses update).
An ERROR like the above means the cluster refused an operation — treat it as the starting point for deeper inspection.
If you see repeated ERRORs on the same object, investigate immediately: check the resource’s events, validate admission webhooks, confirm immutable-field errors, and inspect kubectl describe and kubectl get -o yaml outputs.
Quick troubleshooting checklist Use this compact table to quickly orient any log-based investigation. How to read operator logs — a practical habit Treat logs like evidence:
  1. Find the object identity (namespace/name or controller-provided tag).
  2. Follow the motion: did the controller notice → check → change → quiet, or did it requeue repeatedly?
  3. Compare generation to resourceVersion to detect external vs. controller-driven changes.
  4. Use DEBUG logging selectively when you need to see the internals.
  5. Connect log lines back to the actual Kubernetes objects (inspect with kubectl).
Example repetition that signals a loop:
By following identity, motion, and intent vs. reality, you can convert noisy log streams into a concise narrative of what the controller did and why. Further references

Watch Video