Operators Controllers Reconcile Like a Pro

We have defined our platform API with CustomResourceDefinitions (CRDs). That gives you a typed API surface — but without a controller, creating a Database custom resource (CR) only stores YAML in etcd. No external database is provisioned, no network is configured, and no status or cleanup is performed. Controllers bridge the declarative specification in the API server to the actual external resources. In this lesson you’ll learn:

Why CRDs are only useful when paired with controllers.
How the reconciliation loop works and its guiding principles.
Controller architecture and how informs, queues, and reconcilers interact.
The canonical Reconcile function pattern and a practical Go example.
How to report status with conditions and observedGeneration.
How to implement finalizers to guarantee cleanup.
How to pick an operator framework.

This lesson focuses on controller patterns and implementation best practices. Use a framework (Kubebuilder, Operator SDK, or Metacontroller) to avoid boilerplate and concentrate on business logic.

Example scenario A team publishes a Database CRD with validation, printer columns, and a clear schema. They advertise: You can now create databases using kubectl apply. A developer tries:

# orders-db.yaml
apiVersion: platform.acme.io/v1
kind: Database
metadata:
  name: orders-db
spec:
  size: medium
  engine: postgresql

$ kubectl apply -f orders-db.yaml
database.platform.acme.io/orders-db created

It looks like a success, but nothing happened behind the scenes: the YAML is persisted in etcd but no provisioning, status updates, or cleanup logic run.

A presentation slide titled "CRDs Alone Are Just etcd Data" showing four issues—No database is created, No status updates, No drift correction, and No cleanup on delete—each with a short explanatory note.

A CRD is the menu; the controller is the kitchen that actually prepares the order.

A presentation slide titled "CRDs Alone Are Just etcd Data" that uses a restaurant analogy to explain Kubernetes concepts. It shows three panels: CRD = Menu (describes what you can order), CR = Order ("I'd like a medium PostgreSQL, please."), and Controller = Kitchen (it actually makes the food).

The reconciliation loop The reconciliation loop is the core concept behind all controllers. It continuously observes the API server and attempts to make reality match the desired state described in CRs. The loop can be summarized into four clear phases:

Watch / Observe — receive notifications (via an Informer) when resources change.
Compare — compute the delta between desired state (CR.spec) and actual state (cluster or external system).
Act — create, update, or delete resources to close the gap.
Update status — write current progress and conditions back to CR.status.

Three principles govern robust controllers:

Level-triggered (not edge-triggered): react to state, not individual events. If an event is missed, the next reconcile sees the current state and fixes it.
Idempotency: repeated reconciles must be safe and not cause duplicate or conflicting actions.
Eventual consistency: the system converges toward the desired state over time; transient failures are retried.

Example: if provisioning takes five minutes, set Progressing or Provisioning in status. Reconciles continue until the database is Ready.

A presentation slide titled "The Reconciliation Loop: Desired vs Actual" showing three key principles — "Level-triggered, not edge-triggered," "Eventual consistency," and "Idempotent operations" — each with a short explanation and a colored triangular icon. The slide is branded © KodeKloud.

Quick reference: reconciliation phases

Phase	Purpose	Example action
Watch / Observe	Detect changes without polling the API directly	Informer receives update
Compare	Determine the delta between CR.spec and reality	Check for missing DB instance
Act	Make changes to converge to desired state	Call cloud API to create DB
Update status	Communicate progress and health	Set `status.conditions` to `Progressing` or `Available`

Controller architecture Controllers are composed of a small stack of cooperating components:

API Server — the source of truth where desired state is persisted.
Informer — watches API server changes and keeps a local cache to reduce API traffic.
Work Queue — deduplicates events, rate-limits retries, and ensures single-running reconcile per resource.
Reconciler — your custom logic that compares and acts on resources.

When the Informer notices a change it enqueues a reconcile request. The Work Queue deduplicates rapid changes so only one reconcile runs for the latest state. The framework supplies resilience (retries/backoff), scalability (multiple workers), and consistency semantics — you implement the Reconciler.

A vertical "Controller Architecture" diagram with four colored, numbered blocks (1–4) on the left linked to labeled components on the right: API Server, Informer, Work Queue, and Reconciler with brief descriptions. The slide is copyright KodeKloud.

A presentation slide titled "Controller Architecture" showing three colored arrows labeled Resilience, Consistency, and Scalability with brief notes explaining each. A footer reads that you write the Reconciler while the framework handles watching, queueing, and retries.

The Reconcile function pattern Reconcile is where you express the desired business logic. A typical Reconcile function follows this pattern:

Fetch the resource from the API server (it might have been deleted).
If metadata.deletionTimestamp is set, run finalizer/cleanup logic and return.
Compare spec to actual state and make necessary actions (create/update/delete).
Update status (conditions, observedGeneration, etc.) via the status subresource.

Here is an idiomatic controller-runtime (Go) example that follows those steps:

package controllers

import (
    "context"

    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    v1 "example.com/platform/api/v1"
)

func (r *DBReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the resource
    db := &v1.Database{}
    if err := r.Get(ctx, req.NamespacedName, db); err != nil {
        if client.IgnoreNotFound(err) != nil {
            return ctrl.Result{}, err
        }
        // Resource not found — nothing to do
        return ctrl.Result{}, nil
    }

    // 2. Check if being deleted (finalizer flow)
    if !db.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, db)
    }

    // 3. Reconcile: business logic to make reality match spec
    if err := r.ensureDatabase(ctx, db); err != nil {
        return ctrl.Result{}, err
    }

    // 4. Update status (use status subresource)
    db.Status.Phase = "Ready"
    if err := r.Status().Update(ctx, db); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Note: function names and signatures vary slightly by framework, but the pattern is the same. Status, conditions, and observedGeneration Status is how controllers communicate state to users and automation. Kubernetes-style conditions provide a concise, extensible structure:

type (e.g., Available, Progressing, Degraded)
status (one of "True", "False", "Unknown")
reason and message for human-readable context
lastTransitionTime for timing
observedGeneration (optional) to indicate which metadata.generation the controller has processed

Example status block:

status:
  observedGeneration: 3
  phase: Ready
  conditions:
  - type: Available
    status: "True"
    reason: DatabaseReady
    message: Database is accepting connections
    lastTransitionTime: "2024-01-15T10:30:00Z"
    observedGeneration: 3
  endpoint: orders-db.prod.svc:5432

Use observedGeneration so consumers can compare metadata.generation with status.observedGeneration to know whether the current status reflects the latest spec.

A presentation slide titled "Status Updates and observedGeneration" with a blue rounded box labeled "observedGeneration Pattern." An arrow points to the note: "Compare metadata.generation with status.observedGeneration to see if changes are processed."

Finalizers and cleanup Finalizers guarantee that external resources are cleaned up before the API server permanently deletes the CR. When a user deletes a CR the API server sets metadata.deletionTimestamp but retains the object while finalizers are present. Your controller must:

Detect deletionTimestamp.
Perform external cleanup (delete cloud resources, credentials, networking).
Remove the finalizer from the CR and update it.
Only then will the API server remove the object from etcd.

A simple finalizer flow:

if !db.DeletionTimestamp.IsZero() {
    if hasFinalizer(db) {
        if err := deleteRDSInstance(ctx, db.Spec.Name); err != nil {
            return ctrl.Result{}, err
        }
        if err := deleteSecrets(ctx, db.Namespace); err != nil {
            return ctrl.Result{}, err
        }
        removeFinalizer(db)
        if err := r.Update(ctx, db); err != nil {
            return ctrl.Result{}, err
        }
    }
    // Stop reconciliation; resource is terminating
    return ctrl.Result{}, nil
}

If cleanup fails (for example, cloud APIs are down), the finalizer will remain and the resource will stay in Terminating state. Implement retries with backoff and robust logging so you can recover or manually resolve stuck objects.

Operator frameworks Most teams leverage a framework rather than writing all plumbing from scratch. Here’s a comparison to help you choose:

Framework	Strengths	Good for
Kubebuilder	Official SIG project, Go-first, scaffolding, CRD generation, webhooks, built-in testing	Production-grade Go operators
Operator SDK	Built on Kubebuilder, adds Helm and Ansible operator types	Teams needing Helm/Ansible support or Red Hat ecosystem
Metacontroller	Lightweight, webhook-based; write reconcile logic as an HTTP webhook in any language	Simple operators, polyglot teams, rapid prototypes

Pick the framework that matches your language skills, testing needs, and integration surface. Key takeaways

CRDs alone are just declarative data stored in etcd; controllers make CRs do work.
Reconciliation must be level-triggered, idempotent, and target eventual consistency.
Use status.conditions and observedGeneration to communicate progress and the reconciliation state.
Finalizers ensure external resources are cleaned up before the CR is removed.
Use a framework (Kubebuilder, Operator SDK, Metacontroller) to avoid boilerplate and follow Kubernetes best practices.

A presentation slide titled "Key Takeaways" with four colorful numbered bullets.The bullets summarize Kubernetes concepts: controllers are needed for CRDs to have effect, reconciliation must be idempotent, status/observedGeneration convey resource state, and finalizers ensure cleanup before deletion.

Course Introduction

Platform Architecture and Infrastructure

GitOps and Continuous Delivery

Platform APIs and Self-Service Capabilities

Observability and Operations

Security and Policy Enforcement

Conclusion

Operators Controllers Reconcile Like a Pro

Watch Video