Skip to main content
Version: 0.x (legacy)

Desired-state contracts

This is the idea the whole product is built around, so it is worth understanding precisely rather than by analogy alone.

Desired vs actual, and who owns which​

  • The centre holds desired state. It is declarative — you describe the end state you want (a vSwitch with these adapters, a vNIC with this VLAN), never a sequence of steps. The centre never imperatively drives a host.
  • The agent on each host continuously compares actual state (what the host is really configured as, read live) against desired state (what it was last told to enforce), and emits idempotent operations to close any gap.
  • The agent also keeps a last-honoured copy of its desired state in a local embedded store. If the centre becomes unreachable, the agent does not stop or fall back to some default — it keeps enforcing exactly what it was last given, and reports itself as running autonomously.

This is what makes Ballast different from a centralised-only model: losing the centre never changes intent on a host, only the ability to change that intent.

Generation and ObservedGeneration​

Every desired-state object carries a Generation, incremented by the centre each time it changes. The agent reports back Status.ObservedGeneration — the generation it has actually reconciled actual state to.

  • Generation == ObservedGeneration — settled; the host matches what was asked of it.
  • Generation > ObservedGeneration — a change is outstanding; the host has not yet caught up, whether because it is mid-reconcile or because it is offline from the centre and reconciling against an older cached copy.

The console's drift indicator is built directly from this pair, not from a generic health check:

Ballast console Orchestration panel for a cluster: 9 of 9 steps at 100%, "Settled at generation 7", every step reading AlreadyConfigured or Updated, and a "Desired vs observed" line reading "Observed state matches desired state at generation 7. Nothing to reconcile."

Every step is individually named and its own outcome shown — AlreadyConfigured for a no-op pass, Updated for one that changed something this pass (here, ISCSIConnected had to be adjusted) — rather than one pass/fail rollup for the whole object.

Drift detection​

Drift is any difference between what the agent finds on the host and what its current desired state says should be there — someone changing a vSwitch by hand in Hyper-V Manager, for instance. The agent detects this on every reconcile pass.

Auto-revert vs approval-gated​

Not every drifted field is corrected the same way. Every resource type falls into one of three tiers:

  • Locked — cluster configuration and membership itself. Not reconfigurable through drift handling at all: no auto-revert, no approval flow. Changes to cluster topology only happen through the centre directly. This is the highest blast-radius category (split-brain, quorum loss, stranded VMs), so it gets no autonomous path whatsoever.
  • Additions — net-new, additive changes, like a new virtual switch or a new VLAN. Low risk, since they don't affect anything already working: these are flagged/notified to the admin but applied without waiting for approval, so ordinary day-to-day operations (a new project needs a VLAN) don't require a gate.
  • Modifications — anything that touches existing, currently-working configuration: VM specs (CPU, memory), disks, and witness configuration. The default posture is to not touch what's already working without asking first. The agent detects the drift, notifies the admin through the centre, and waits for an explicit approve or reject before applying it. Validation runs as part of applying an approved change, and a failed validation leaves the previous working configuration untouched. Everything not in the first two tiers auto-reverts to the desired-state contract on drift; this tier is the deliberate exception because the cost of a wrong auto-revert (or a wrong auto-apply) is high enough to warrant a human in the loop.

Witness configuration changes get an extra safeguard: because a broken witness is a silent failure mode (no symptoms until an actual node failure), approving a witness change tests both the old and the new witness, not just the new one — which can also surface that the current witness is already broken, independently of the change being reviewed.

Idempotency, non-negotiably​

Applying the same desired state twice must always be a no-op. Every reconciler is written and tested to this rule; it is what makes the reconcile loop safe to run continuously rather than only on a triggered change.