Async Reconciliation in Kubernetes

Kubernetes doesn’t try to make changes happen immediately and perfectly. Instead, it continuously reconciles what is with what should be.

You declare your desired state: “I want 3 replicas of this pod.” Kubernetes stores that. Then controllers - running in loops - compare actual state to desired state and take action to close the gap.

while true:
    actual = get_current_state()
    desired = get_desired_state()
    if actual != desired:
        take_action_to_reconcile()
    sleep(interval)

This is the reconciliation loop. It’s simple, but the pattern is everywhere in Kubernetes.

The key insight: instead of trying to execute a perfect sequence of operations (create this, then that, handle this failure), you just keep nudging reality toward the goal. If something fails, the next loop iteration will try again. If state drifts, the loop will notice and fix it.

This makes the system self-healing by default. Kill a pod? The ReplicaSet controller notices the count is wrong and creates a new one. Node goes down? Controllers notice their pods are missing and reschedule them elsewhere.

It also means Kubernetes is eventually consistent. When you apply a change, you’re not waiting for it to complete - you’re declaring intent. The system will get there, but not instantly.

The downside: debugging can be tricky. There’s no single place where “the action” happens. Controllers run independently, events cascade, and you need to understand the whole reconciliation model to follow what’s going on.