Reconciliation Loop

How reconciliation loops work

The pattern: observe actual state, compare to desired state, take action to close the gap. Repeat forever. This is the core of how Kubernetes controllers work — and it's surprisingly robust.

Level-triggered vs edge-triggered

The controller doesn't react to events ("a pod crashed"). It periodically asks "is the world the way I want it?" and fixes any drift. This means it handles any kind of failure — crashes, misconfigurations, manual changes — the same way. It doesn't need to know why things are wrong, just that they're wrong.

Action latency

Creating a pod isn't instant. The container image needs to pull, the process needs to start, health checks need to pass. During this time, the controller needs to account for "in-flight" work — if 2 pods are already starting, don't create 2 more. This is why the controller counts running + starting, not just running.

Why it converges

Try changing the desired count and watch. The controller creates/terminates pods to close the gap. Even if you crash half the pods, the next reconcile cycle notices and starts replacements. The system is self-healing — not because it's clever, but because it keeps checking and correcting.

Max surge and rate limiting

Without a limit on concurrent starts, a big scale-up would try to start 10 pods at once, overwhelming the system. Max surge limits how many can be starting simultaneously. Increase the reconcile interval to see how the system still converges, just more slowly — proving that level-triggered works even with infrequent checks.

Where you'll find this

Kubernetes controllers (Deployment, ReplicaSet, StatefulSet, DaemonSet), Terraform plan/apply, Puppet/Chef convergence, AWS Auto Scaling Groups, systemd service restarts, React's reconciliation algorithm (virtual DOM diff), database replica repair — the pattern is everywhere.