> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Operators Controllers Reconcile Like a Pro > Explains Kubernetes controllers and reconciliation patterns for CRDs, status observation, finalizers, and selecting operator frameworks. We have defined our platform API with [CustomResourceDefinitions (CRDs)](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/). That gives you a typed API surface — but without a controller, creating a Database custom resource (CR) only stores YAML in etcd. No external database is provisioned, no network is configured, and no status or cleanup is performed. Controllers bridge the declarative specification in the API server to the actual external resources. In this lesson you'll learn: * Why CRDs are only useful when paired with controllers. * How the reconciliation loop works and its guiding principles. * Controller architecture and how informs, queues, and reconcilers interact. * The canonical Reconcile function pattern and a practical Go example. * How to report status with conditions and `observedGeneration`. * How to implement finalizers to guarantee cleanup. * How to pick an operator framework. This lesson focuses on controller patterns and implementation best practices. Use a framework (Kubebuilder, Operator SDK, or Metacontroller) to avoid boilerplate and concentrate on business logic. Example scenario A team publishes a Database CRD with validation, printer columns, and a clear schema. They advertise: You can now create databases using `kubectl apply`. A developer tries: ```yaml theme={null} # orders-db.yaml apiVersion: platform.acme.io/v1 kind: Database metadata: name: orders-db spec: size: medium engine: postgresql ``` ```console theme={null} $ kubectl apply -f orders-db.yaml database.platform.acme.io/orders-db created ``` It looks like a success, but nothing happened behind the scenes: the YAML is persisted in etcd but no provisioning, status updates, or cleanup logic run. ![A presentation slide titled "CRDs Alone Are Just etcd Data" showing four issues—No database is created, No status updates, No drift correction, and No cleanup on delete—each with a short explanatory note.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/crds-alone-etcd-data-issues.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=6729363e750940f5c53bf3bfe1df4dc2) A CRD is the menu; the controller is the kitchen that actually prepares the order. ![A presentation slide titled "CRDs Alone Are Just etcd Data" that uses a restaurant analogy to explain Kubernetes concepts. It shows three panels: CRD = Menu (describes what you can order), CR = Order ("I'd like a medium PostgreSQL, please."), and Controller = Kitchen (it actually makes the food).](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/crds-etcd-restaurant-analogy.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=0b2dbf5353edf88433457ec5a6a5f130) The reconciliation loop The reconciliation loop is the core concept behind all controllers. It continuously observes the API server and attempts to make reality match the desired state described in CRs. The loop can be summarized into four clear phases: 1. Watch / Observe — receive notifications (via an Informer) when resources change. 2. Compare — compute the delta between desired state (CR.spec) and actual state (cluster or external system). 3. Act — create, update, or delete resources to close the gap. 4. Update status — write current progress and conditions back to CR.status. Three principles govern robust controllers: * Level-triggered (not edge-triggered): react to state, not individual events. If an event is missed, the next reconcile sees the current state and fixes it. * Idempotency: repeated reconciles must be safe and not cause duplicate or conflicting actions. * Eventual consistency: the system converges toward the desired state over time; transient failures are retried. Example: if provisioning takes five minutes, set `Progressing` or `Provisioning` in status. Reconciles continue until the database is `Ready`. ![A presentation slide titled "The Reconciliation Loop: Desired vs Actual" showing three key principles — "Level-triggered, not edge-triggered," "Eventual consistency," and "Idempotent operations" — each with a short explanation and a colored triangular icon. The slide is branded © KodeKloud.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/reconciliation-loop-desired-vs-actual.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=4261aeef96158c03f628db9c52c6621a) Quick reference: reconciliation phases | Phase | Purpose | Example action | | - | - | - | | Watch / Observe | Detect changes without polling the API directly | Informer receives update | | Compare | Determine the delta between CR.spec and reality | Check for missing DB instance | | Act | Make changes to converge to desired state | Call cloud API to create DB | | Update status | Communicate progress and health | Set `status.conditions` to `Progressing` or `Available` | Controller architecture Controllers are composed of a small stack of cooperating components: * API Server — the source of truth where desired state is persisted. * Informer — watches API server changes and keeps a local cache to reduce API traffic. * Work Queue — deduplicates events, rate-limits retries, and ensures single-running reconcile per resource. * Reconciler — your custom logic that compares and acts on resources. When the Informer notices a change it enqueues a reconcile request. The Work Queue deduplicates rapid changes so only one reconcile runs for the latest state. The framework supplies resilience (retries/backoff), scalability (multiple workers), and consistency semantics — you implement the Reconciler. ![A vertical "Controller Architecture" diagram with four colored, numbered blocks (1–4) on the left linked to labeled components on the right: API Server, Informer, Work Queue, and Reconciler with brief descriptions. The slide is copyright KodeKloud.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/controller-architecture-api-informer-workqueue-reconciler.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=85d26c6b439e3f43c583ac2f23d5a054) ![A presentation slide titled "Controller Architecture" showing three colored arrows labeled Resilience, Consistency, and Scalability with brief notes explaining each. A footer reads that you write the Reconciler while the framework handles watching, queueing, and retries.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/controller-architecture-resilience-consistency-scalability.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=cad32d8eed00be98d89bb64148c5c6f2) The Reconcile function pattern Reconcile is where you express the desired business logic. A typical Reconcile function follows this pattern: 1. Fetch the resource from the API server (it might have been deleted). 2. If `metadata.deletionTimestamp` is set, run finalizer/cleanup logic and return. 3. Compare spec to actual state and make necessary actions (create/update/delete). 4. Update `status` (conditions, `observedGeneration`, etc.) via the status subresource. Here is an idiomatic controller-runtime (Go) example that follows those steps: ```go theme={null} package controllers import ( "context" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" v1 "example.com/platform/api/v1" ) func (r *DBReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // 1. Fetch the resource db := &v1.Database{} if err := r.Get(ctx, req.NamespacedName, db); err != nil { if client.IgnoreNotFound(err) != nil { return ctrl.Result{}, err } // Resource not found — nothing to do return ctrl.Result{}, nil } // 2. Check if being deleted (finalizer flow) if !db.DeletionTimestamp.IsZero() { return r.handleDeletion(ctx, db) } // 3. Reconcile: business logic to make reality match spec if err := r.ensureDatabase(ctx, db); err != nil { return ctrl.Result{}, err } // 4. Update status (use status subresource) db.Status.Phase = "Ready" if err := r.Status().Update(ctx, db); err != nil { return ctrl.Result{}, err } return ctrl.Result{}, nil } ``` Note: function names and signatures vary slightly by framework, but the pattern is the same. Status, conditions, and observedGeneration Status is how controllers communicate state to users and automation. Kubernetes-style conditions provide a concise, extensible structure: * `type` (e.g., `Available`, `Progressing`, `Degraded`) * `status` (one of `"True"`, `"False"`, `"Unknown"`) * `reason` and `message` for human-readable context * `lastTransitionTime` for timing * `observedGeneration` (optional) to indicate which `metadata.generation` the controller has processed Example status block: ```yaml theme={null} status: observedGeneration: 3 phase: Ready conditions: - type: Available status: "True" reason: DatabaseReady message: Database is accepting connections lastTransitionTime: "2024-01-15T10:30:00Z" observedGeneration: 3 endpoint: orders-db.prod.svc:5432 ``` Use `observedGeneration` so consumers can compare `metadata.generation` with `status.observedGeneration` to know whether the current `status` reflects the latest `spec`. ![A presentation slide titled "Status Updates and observedGeneration" with a blue rounded box labeled "observedGeneration Pattern." An arrow points to the note: "Compare metadata.generation with status.observedGeneration to see if changes are processed."](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/observed-generation-pattern-status-updates.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=09fbcdb6b617e2c0b6d39ebc8ccfda37) Finalizers and cleanup Finalizers guarantee that external resources are cleaned up before the API server permanently deletes the CR. When a user deletes a CR the API server sets `metadata.deletionTimestamp` but retains the object while finalizers are present. Your controller must: * Detect `deletionTimestamp`. * Perform external cleanup (delete cloud resources, credentials, networking). * Remove the finalizer from the CR and update it. * Only then will the API server remove the object from etcd. A simple finalizer flow: ```go theme={null} if !db.DeletionTimestamp.IsZero() { if hasFinalizer(db) { if err := deleteRDSInstance(ctx, db.Spec.Name); err != nil { return ctrl.Result{}, err } if err := deleteSecrets(ctx, db.Namespace); err != nil { return ctrl.Result{}, err } removeFinalizer(db) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } } // Stop reconciliation; resource is terminating return ctrl.Result{}, nil } ``` If cleanup fails (for example, cloud APIs are down), the finalizer will remain and the resource will stay in `Terminating` state. Implement retries with backoff and robust logging so you can recover or manually resolve stuck objects. Operator frameworks Most teams leverage a framework rather than writing all plumbing from scratch. Here’s a comparison to help you choose: | Framework | Strengths | Good for | | - | - | - | | [Kubebuilder](https://book.kubebuilder.io/) | Official SIG project, Go-first, scaffolding, CRD generation, webhooks, built-in testing | Production-grade Go operators | | [Operator SDK](https://sdk.operatorframework.io/) | Built on Kubebuilder, adds Helm and Ansible operator types | Teams needing Helm/Ansible support or Red Hat ecosystem | | [Metacontroller](https://metacontroller.github.io/) | Lightweight, webhook-based; write reconcile logic as an HTTP webhook in any language | Simple operators, polyglot teams, rapid prototypes | Pick the framework that matches your language skills, testing needs, and integration surface. Key takeaways * CRDs alone are just declarative data stored in etcd; controllers make CRs do work. * Reconciliation must be level-triggered, idempotent, and target eventual consistency. * Use `status.conditions` and `observedGeneration` to communicate progress and the reconciliation state. * Finalizers ensure external resources are cleaned up before the CR is removed. * Use a framework (Kubebuilder, Operator SDK, Metacontroller) to avoid boilerplate and follow Kubernetes best practices. ![A presentation slide titled "Key Takeaways" with four colorful numbered bullets.The bullets summarize Kubernetes concepts: controllers are needed for CRDs to have effect, reconciliation must be idempotent, status/observedGeneration convey resource state, and finalizers ensure cleanup before deletion.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Operators-Controllers-Reconcile-Like-a-Pro/k8s-key-takeaways-controllers-idempotent-finalizers.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=c8ad910ddbf0fe70d1f1191d5c0b0c15) Further reading and references * [Kubernetes CustomResourceDefinitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) * [controller-runtime (controller pattern and helpers)](https://github.com/kubernetes-sigs/controller-runtime) * [Kubebuilder book](https://book.kubebuilder.io/) * [Operator SDK](https://sdk.operatorframework.io/) * [Metacontroller documentation](https://metacontroller.github.io/) * Conditions guidance: [API conventions — conditions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-conditions) > **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/756ffaae-767b-4743-9724-c05d3fbf9a18/lesson/4c8d6b72-361f-4c1c-8b62-cf84a251000f)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.