Operators Controllers Reconcile Like a Pro

We can define platform APIs with CustomResourceDefinitions (CRDs) so users create custom resources (CRs). That validates YAML with the API server, but validation alone doesn’t make things happen. If you create a Database CR today, nothing will be provisioned automatically — the YAML just sits in etcd until a controller acts on it. Controllers are the missing piece that bridge declared intent to real-world resources. In this lesson we will:

Explain why CRDs are useful only when paired with controllers.
Describe the reconciliation loop and its governing principles.
Describe controller architecture and the typical reconcile function shape.
Cover status reporting with conditions and observedGeneration.
Implement finalizers for cleanup on deletion.
Compare common operator frameworks so you can pick the right one.

The image outlines learning objectives related to CRDs and controller architecture, including understanding CRDs' need for controllers and explaining the reconciliation loop.

This is a longer lesson because controllers are one of the most fundamental concepts in Kubernetes. Deployments, StatefulSets, Services, and every custom operator all depend on controllers to make the desired state real. Scenario: a shiny Database CRD A platform team builds a Database CRD with enum validations, printer columns, and a solid OpenAPI schema. They tell developers: create a database with kubectl apply.

The image illustrates a concept where a platform team interacts with a database CRD (Custom Resource Definition) through processes like enum validation, printer columns, and schema management.

The developer applies a YAML:

The image shows a diagram with a "Platform Team" informing "Developers" that they can now create databases using kubectl apply.

apiVersion: platform.acme.io/v1
kind: Database
metadata:
  name: orders-db
spec:
  size: medium
  engine: postgresql

Command:

kubectl apply -f orders-db.yaml
# Output
database.platform.acme.io/orders-db created

It appears successful, but there is still no PostgreSQL instance, no network configuration, and no status updates. Without a controller, the CR is only persisted in etcd — nothing reconciles it to reality.

The image is a diagram highlighting the limitations of CRDs as etcd data, including issues like no database creation, lack of status updates, absence of drift correction, and no cleanup on delete.

Controller = the software that makes desired state real. A helpful analogy:

The image explains the roles of CRD (Menu), CR (Order), and Controller (Kitchen) in a Kubernetes context, using a restaurant analogy.

CRD is the menu: it defines what can be requested.
CR (the object you apply) is the order you place.
Controller is the kitchen: it actually prepares the meal and reports progress.

The Reconciliation Loop Reconciliation is the core control logic controllers run repeatedly to converge actual state to desired state. Typical phases:

Watch / observe — get notifications for creates/updates/deletes.
Compare — compute desired state (from spec) vs actual state (external resources, cloud APIs, cluster objects).
Act — create/update/delete resources to close the gap.
Report — update the CR’s status (conditions, observedGeneration, phase).

Examples for a Database CR:

If the external DB does not exist, create it.
If the size changed, resize the DB.
If the CR is deleted, perform cleanup.
Update status to show provisioning progress or readiness.

Follow three guiding principles for safe and robust reconcilers:

Level-triggered: react to current state, not transient events. If an event is missed, the next reconcile still inspects state and corrects drift.
Idempotency: running Reconcile multiple times should produce the same final result as running it once—always check current state first.
Eventual consistency: the system can take multiple reconciles to converge; expect retries and backoff.

The image outlines the key principles of "The Reconciliation Loop: Desired vs Actual," emphasizing level-triggered processes, eventual consistency, and idempotent operations. It suggests reconciling based on current state rather than events, allowing systems to converge over time and handle repeated operations with consistent results.

Use status to inform users: a five-minute provisioning operation can report Progressing while the controller continues to reconcile and later update Ready. Controller internals: four collaborating layers Most operator frameworks implement this architecture for you; your job is to implement the Reconciler (the business logic).

API server — source of truth and where desired state is stored.
Informer — watches API server and maintains a local cache to reduce API traffic.
Work queue — the informer enqueues resource keys; queue deduplicates events so the reconciler runs for the latest state.
Reconciler — your implementation: read resource, compare desired vs actual, act, and return a result (requeue/no-requeue/error).

The image is a diagram of a controller architecture consisting of three components: API Server, Informer, and Work Queue, each with specific roles described alongside.

The image illustrates a controller architecture composed of four components: an API Server, an Informer, a Work Queue, and a Reconciler, each with specific functions outlined.

The framework handles watching, queuing, retries, and worker scaling so you can focus on making the reconciler idempotent and correct.

The image illustrates a "Controller Architecture" with focus on resilience, consistency, and scalability, explaining the benefits of the design. It highlights that the framework manages watching, queuing, and retries, while the user writes the reconciler.

Reconcile function pattern (conceptual Go example) You don’t need to memorize Go—focus on the flow. This example mirrors the signature used by controller-runtime: Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error).

// Reconcile compares desired and actual state and acts to converge them.
// Signature in controller-runtime: Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
func (r *DBReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the resource
    db := &platformv1.Database{}
    if err := r.Get(ctx, req.NamespacedName, db); err != nil {
        if apierrors.IsNotFound(err) {
            // Resource deleted after request — nothing to do.
            return ctrl.Result{}, nil
        }
        // Transient error reading the object — requeue.
        return ctrl.Result{}, err
    }

    // 2. Check if being deleted (finalizer handling)
    if db.GetDeletionTimestamp() != nil {
        return r.handleDeletion(ctx, db)
    }

    // 3. Main reconcile logic (idempotent): ensure external DB exists and matches spec
    if err := r.ensureDatabase(ctx, db); err != nil {
        // If transient, requeue with backoff
        return ctrl.Result{}, err
    }

    // 4. Update status: observedGeneration, conditions, phase, etc.
    db.Status.ObservedGeneration = db.Generation
    db.Status.Phase = "Ready"
    // set conditions as appropriate...
    if err := r.Status().Update(ctx, db); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Best practices for your reconcile function:

Always fetch the latest object from the API server or cache.
Handle deletion via finalizers and short-circuit if the object is gone.
Make operations idempotent — check external state before making changes.
Use the status subresource to update fields like observedGeneration and conditions.

Status and conditions Status is how your controller reports progress and health. Kubernetes conditions are a well-established pattern: each condition includes type, status (True/False/Unknown), reason, and message. Common condition types:

Available / Ready — resource is ready to serve.
Progressing — creation/update is in progress.
Degraded — running but not operating correctly.

observedGeneration pattern When a user modifies the spec, Kubernetes increments metadata.generation. Your controller sets status.observedGeneration to the generation it last reconciled. Comparing metadata.generation and status.observedGeneration tells users whether the controller has acted on the latest change.

The image explains the "observedGeneration" pattern, where you compare metadata.generation with status.observedGeneration to check if changes are processed.

Finalizers and deletion flow Finalizers allow controllers to perform cleanup before Kubernetes permanently removes a CR. When a user deletes a CR, the API server sets deletionTimestamp but keeps the object until finalizers are cleared—this is the controller’s opportunity to delete external resources (cloud instances, secrets, networks). Typical finalizer handling:

// This runs when the CR has a deletion timestamp set.
func (r *DBReconciler) handleDeletion(ctx context.Context, db *platformv1.Database) (ctrl.Result, error) {
    if hasFinalizer(db, dbFinalizer) {
        // Perform cleanup of external resources.
        if err := deleteRDSInstance(ctx, db.Spec.Name); err != nil {
            // Retry on transient errors; log and requeue.
            return ctrl.Result{}, err
        }
        if err := deleteSecrets(ctx, db.Namespace, db.Name); err != nil {
            return ctrl.Result{}, err
        }

        // Remove finalizer and update the object to allow deletion.
        removeFinalizer(db, dbFinalizer)
        if err := r.Update(ctx, db); err != nil {
            return ctrl.Result{}, err
        }
    }
    // Nothing else to do; let API server delete the object.
    return ctrl.Result{}, nil
}

Deletion flow summary:

User runs kubectl delete.
API server sets deletionTimestamp and retains the object.
Controller observes the timestamp, performs cleanup, and removes finalizers.
When no finalizers remain, the API server deletes the object.

If cleanup fails (for example, cloud API outages), the object will remain in a Terminating state while finalizers persist. Implement robust retries, exponential backoff, and comprehensive logging so cleanup eventually completes and you avoid orphaned resources.

Operator frameworks Most teams avoid writing controller boilerplate from scratch. Common frameworks let you focus on business logic:

The image is a comparison chart of three operator frameworks: Kubebuilder, Operator SDK, and Metacontroller, highlighting their features and best use cases.

Below is a concise comparison to help you choose:

Framework	Language / Style	Best for	Notes & Links
Kubebuilder	Go (code-first)	Production Go operators	`https://book.kubebuilder.io/` — scaffolds controllers, CRDs, webhooks, testing
Operator SDK	Go / Helm / Ansible	Multi-language teams, Red Hat ecosystem	Built on Kubebuilder; supports Helm & Ansible operators — `https://sdk.operatorframework.io/`
Metacontroller	Any (webhook)	Lightweight, polyglot operators	Implement reconcile as an HTTP server — `https://metacontroller.github.io/`

Key takeaways

CRDs without controllers are just stored data — controllers make the desired state become real.
Design your Reconcile to be idempotent and level-triggered: always read current state and act only when needed.
Use status, conditions, and observedGeneration to transparently communicate what the controller has processed.
Implement finalizers to clean up external resources; handle errors with retries and backoff to avoid stuck Terminating objects.

Additional resources

Kubernetes controllers & reconciliation concepts: https://kubernetes.io/docs/concepts/architecture/controller/
controller-runtime (Reconcile signature and helpers): https://github.com/kubernetes-sigs/controller-runtime
Kubebuilder book: https://book.kubebuilder.io/
Operator SDK docs: https://sdk.operatorframework.io/
Metacontroller: https://metacontroller.github.io/

Course Introduction

Platform Architecture and Infrastructure

GitOps and Continuous Delivery

Platform APIs and Self-Service Capabilities

Observability and Operations

Security and Policy Enforcement

Conclusion

Operators Controllers Reconcile Like a Pro

Watch Video