Back up a cluster

The operator creates a transient Job for each due or manually requested backup. The Job authenticates to OpenBao, streams a Raft snapshot to object storage, and records the result on status.backup. Snapshot bytes do not pass through the controller.

The stored object is a standard OpenBao Raft snapshot. Recovery can use an operator-managed restore or a manual restore into a separately provisioned cluster. An operator installation is not required for the manual path.

Before you begin

Prepare two independent identities:

1. An OpenBao identity that can read sys/storage/raft/snapshot. Prefer a JWT role bound to the generated <cluster>-backup-serviceaccount; use tokenSecretRef only as a compatibility path.
2. An object-storage identity. Use credentialsSecretRef, workload-identity metadata, or S3 s3.roleArn.

When self-init OIDC bootstrap is enabled, an empty jwtAuthRole resolves to openbao-operator-backup and the operator creates the policy and role during initial bootstrap. Otherwise, configure jwtAuthRole or tokenSecretRef explicitly.

Hardened clusters also require explicit, port-scoped spec.network.egressRules and an explicit storage identity. See Configure network policy.

Configure an S3 backup

This is the preferred minimal shape when self-init OIDC bootstrap is enabled:

spec:
  backup:
    schedule: "0 3 * * *"
    target:
      provider: s3
      endpoint: https://s3.amazonaws.com
      bucket: openbao-backups
      s3:
        region: us-east-1
      pathPrefix: clusters
      credentialsSecretRef:
        name: s3-backup-credentials
    retention:
      maxCount: 14
      maxAge: 336h

schedule is a five-field cron expression. The default provider is s3, but set it explicitly so intent remains clear.

The operator persists the first scheduled run in status.backup.nextScheduledBackup before it waits for that run. The deadline survives reconciliation and controller restarts. Check status.backup.lastBackupTime and status.backup.lastBackupName to verify that a scheduled backup succeeds.

bucket is required for every provider. Azure uses it as the container name. The existing azure.container field remains a compatibility alias; a nonempty value takes precedence over bucket. Use bucket alone in new manifests.

For S3, group region, usePathStyle, and roleArn under target.s3. Each configured nested field overrides its top-level compatibility field. Omitted nested fields use the top-level value, including its existing CRD default. The nested block has no defaults: s3: {} preserves the compatibility values, s3.usePathStyle: false overrides top-level true, and s3.roleArn: "" disables the top-level role path. An empty s3.region uses us-east-1. Existing top-level S3 manifests remain supported. The s3 block is accepted only for provider: s3.

For Azure, target.endpoint is the complete Blob service URL. The operator uses it without adding the storage account name. Omit it to use https://<storageAccount>.blob.core.windows.net, including when private DNS resolves that hostname to a private endpoint. For Azurite, include the account path, for example http://azurite.azure.svc.cluster.local:10000/devstoreaccount1. A credential Secret’s connectionString supplies its own endpoint and takes precedence over target.endpoint.

For a storage endpoint signed by a private certificate authority, add its PEM certificate bundle under caCert in the referenced storage Secret. S3, GCS, and Azure backup and restore executors add this bundle to the system trust roots. The bundle is separate from the OpenBao API CA. Keep insecureSkipVerify disabled.

The Secret can contain only caCert when authentication uses workload identity or the provider credential chain. Adding storage trust does not select static credentials; retain the required role or workload-identity configuration.

For S3, a nonempty region entry in the storage Secret takes precedence over the target region for backup, restore, and controller-side storage operations. An absent or blank entry uses the configured target region, then us-east-1 when no region is configured. A region-only Secret does not select static authentication; the provider credential chain still applies.

For platform workload identity, set target.workloadIdentity.serviceAccountAnnotations and, when required, target.workloadIdentity.podLabels. For S3 web identity, the effective target.s3.roleArn or compatibility target.roleArn projects an STS token and configures the Job explicitly.

Use a static OpenBao token only when needed

If JWT auth is unavailable, create a same-namespace Secret whose token value can read the Raft snapshot endpoint, then reference it:

spec:
  backup:
    tokenSecretRef:
      name: openbao-backup-token

jwtAuthRole takes precedence when both fields are present. Rotate and revoke a static token as a high-sensitivity credential.

Delegate configuration safely

The identity that changes the OpenBaoCluster needs:

• get on each referenced Secret
• usecloudidentities on the cluster when it sets an effective S3 role or workload-identity metadata
• usecustomexecutables on the cluster when it overrides spec.backup.image (usehelperimages remains a compatibility alias)

Admission rejects loopback and link-local storage endpoints. Hardened clusters also reject insecure TLS and ambient storage identity.

Storage clients reject AWS metadata (fd00:ec2::254), EKS Pod Identity (fd00:ec2::23), and Google Cloud metadata (fd20:ce::254) as storage destinations, including DNS answers and redirects. Other private IPv6 storage addresses remain supported. Credential discovery uses a separate HTTP client, so these destination checks do not block workload-identity providers.

Trigger and verify a backup

The operator does not create a CronJob. Trigger an immediate backup by changing the one-shot annotation value:

kubectl -n <namespace> annotate openbaocluster <name> \
  openbao.org/trigger-backup="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

The controller clears the annotation after accepting the request. It skips a duplicate request while another backup Job is active.

kubectl -n <namespace> get openbaocluster <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="BackupConfigurationReady")]}{.status}{" "}{.reason}{"\n"}{.message}{"\n"}{end}'
kubectl -n <namespace> get openbaocluster <name> -o jsonpath='{.status.backup}' | jq
kubectl -n <namespace> get jobs -l openbao.org/cluster=<name>

A successful run advances lastBackupTime, lastBackupName, lastBackupSize, and lastBackupDuration. Failure details are in lastFailureReason, lastFailureMessage, and lastFailureTime. Confirm the object independently in storage before relying on it.

If the snapshot stream fails during upload, the executor aborts the write, so the failed snapshot creates no object. After upload, it checks that the stored object size matches the number of bytes streamed and deletes the object if they differ. That deletion can fail on object-lock or write-only buckets, so check for leftover objects after a failed run.

Set retention and pre-upgrade snapshots

maxCount: 0 and an empty maxAge mean unlimited retention. The operator applies retention after a successful upload and never turns a retention error into a failed snapshot. Retention always keeps the newest snapshot and the snapshot recorded in status.backup.lastBackupName, even when they are older than maxAge. Snapshot object keys use the time the backup Job was created, so a backup delayed past its schedule is not treated as old.

Retention only manages scheduled and manual snapshots stored directly under the cluster prefix. It leaves pre-upgrade and blue-green phase snapshots, nested objects, and any other files in that prefix untouched. Remove those by hand or with storage lifecycle rules.

Retention runs in a separate Job with the successful backup’s writer ServiceAccount and storage credentials. This supports credential Secrets, S3 web identity, and GCS or Azure workload identity. The storage identity needs permission to list and delete snapshots in addition to the permissions needed to upload and verify them.

The operator records snapshot success before it reserves retention. It holds the backup operation lock during retention, which can delay the next backup or upgrade. New restore-test reservations wait for retention; an existing restore test’s snapshot remains protected. The worker has an absolute four-minute deadline, and the Job has a five-minute active deadline. A delayed start does not extend the worker deadline. There is one Create attempt per successful source Job; retention failure does not change the snapshot result. The next successful backup can retry.

Check status.backup.retention.active, status.backup.retention.last, and the RetentionSucceeded condition under status.backup.retention.conditions. Succeeded, Failed, and Skipped refer to cleanup, independently of status.backup.lastBackupName. A failed batch can have partial effects; unknown partial deletions are not counted as successful deletions. The worker uses the successful backup’s frozen target and configuration generation. A configuration change before submission skips that attempt. A submitted worker finishes with its frozen plan.

Enable spec.upgrade.preUpgradeSnapshot: true to require a snapshot before either upgrade strategy. Blue-green also accepts spec.upgrade.blueGreen.preUpgradeSnapshot, but the top-level field keeps one policy across strategies. The upgrade does not continue if its required snapshot fails.

A backup is complete only after you restore it into an isolated target and validate the recovered data. Schedule restore tests to do this automatically for the latest backup.