Upgrade a cluster

Use RollingUpdate unless you need parallel validation or a manual traffic cutover. Both strategies require a healthy cluster, a supported target version, and a working upgrade identity.

Choose the strategy

Check the compatibility matrix and the upstream release notes before choosing the target.

Prepare the rollout

Enable a mandatory recovery point for the next change:

spec:
  upgrade:
    preUpgradeSnapshot: true

This requires a valid spec.backup. The operator acquires an operation lock so an upgrade, backup, and restore do not perform conflicting long-running work.

Change strategies on an existing cluster

Change only the strategy, then wait for the operator to accept it. Do not combine the strategy transition with a version, image, replica, storage, or restart change.

kubectl -n <namespace> patch openbaocluster <name> --type merge -p '{
  "spec": {
    "upgrade": {
      "strategy": "RollingUpdate"
    }
  }
}'

kubectl -n <namespace> get openbaocluster <name> \
  -o jsonpath='{.status.acceptedUpgradeStrategy}{"\n"}'

Before switching to BlueGreen, configure spec.upgrade.jwtAuthRole or use the default openbao-operator-upgrade role created by self-init OIDC bootstrap. Without policy reconciliation, a role created for a rolling-only cluster might need its policy expanded before the switch; self-init requests are not replayed later.

With approved policy reconciliation, enrollment approves both supported strategies. The operator updates the built-in upgrade policy after a strategy change, without another approval. Wait for PolicyReconciliationReady=True as well as strategy acceptance before requesting the version change. Custom policies attached to a custom upgrade role remain administrator-managed.

Request the version change

After the strategy is accepted, change spec.version in a separate request:

kubectl -n <namespace> patch openbaocluster <name> --type merge -p '{
  "spec": {
    "version": "<target-version>"
  }
}'

The version must be semantic version syntax. The operator blocks downgrades below status.currentVersion. If you set spec.image, keep a semantic tag aligned with spec.version; digest-pinned images still use spec.version as the upgrade intent.

Control a blue-green promotion

Set spec.upgrade.blueGreen.autoPromote: false before the upgrade begins to hold a healthy Green revision in Syncing. Approve it by changing the one-shot request value:

kubectl -n <namespace> patch openbaocluster <name> --type merge -p "{
  \"spec\": {
    \"upgrade\": {
      \"requests\": {
        \"promote\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
      }
    }
  }
}"

autoPromote is snapshotted when an upgrade starts. Changing it during an in-flight upgrade affects only the next upgrade.

When a BlueGreen validation hook is configured, the operator verifies its image through spec.operatorImageVerification before creating the Job and pins a successful result by digest. Block stops the upgrade when verification fails; Warn keeps the original reference and is not permitted by the Hardened profile. The Job disables ServiceAccount token automounting, runs non-root with a read-only root filesystem, drops all capabilities, uses RuntimeDefault seccomp, inherits spec.imagePullSecrets, and receives bounded resources.

Recover a held rolling failure

Fix the cause recorded in status.upgrade.failure, then change spec.upgrade.requests.retry:

kubectl -n <namespace> patch openbaocluster <name> --type merge -p "{
  \"spec\": {
    \"upgrade\": {
      \"requests\": {
        \"retry\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
      }
    }
  }
}"

Before resetting failed Pods, the operator waits for the StatefulSet to reflect the current cluster generation and for the StatefulSet controller to observe that template. The Bao container image must match the resolved image recorded by workload reconciliation, including a verified digest when image verification succeeds. If the image differs or the resolved-image annotation is missing, the retry waits for workload reconciliation.

The operator records each rolling attempt in status.upgrade.executionID. Controller restarts preserve this value. A new upgrade or accepted retry uses a new value, so retained leader step-down Jobs cannot satisfy a later attempt. Before accepting a retry or changing the target version, the operator waits for the previous step-down Job and its Pods to terminate. Progress created by an older operator can resume with its existing Jobs only when their ownership, action, and creation time identify the current attempt. If that evidence is missing, the operator holds the upgrade; inspect status.upgrade.failure, resolve the cause, and submit a new retry request.

Each request is edge-triggered: use a new non-empty value. For blue-green rollback repair failures, use Recover a failed rollback instead of retrying blindly.

A blue-green upgrade can only roll back before the Cleanup phase, because Cleanup removes Blue peers from Raft. From Cleanup onward, the operator refuses spec.upgrade.requests.rollback and automatic rollback with a RollbackRefused warning event. It retries the current phase, and applies any spec change made during that time after the current upgrade completes. The operator records Green’s target image, version, and replica count and Blue’s original replica count in status.blueGreen. Changes to spec.replicas do not change the active upgrade’s cleanup checks or rollback quorum. When resuming an older operation without these fields, the operator recovers its image and replica counts from the existing StatefulSets; it stops if those inputs cannot be recovered.

Before deleting a retired Blue or aborted Green StatefulSet, the operator verifies its current-cluster controller owner reference, openbao.org/owner-uid annotation, and Pod-template revision. An ownership or revision mismatch stops cleanup and preserves the StatefulSet. For an unrevisioned Blue workload, the operator also checks status.blueGreen.blueControllerRevision when that field is recorded.

Verify the result

kubectl -n <namespace> get openbaocluster <name> -w
kubectl -n <namespace> get pods -l openbao.org/cluster=<name> -o wide
kubectl -n <namespace> get events --sort-by=.lastTimestamp

Finish only when status.currentVersion equals spec.version, status.phase=Running, Available=True, all declared replicas are Ready, and Raft membership matches the intended topology.