Test a change

Start at the lowest-cost layer that can prove the contract. Move upward when the behavior depends on Kubernetes API semantics, controller wiring, or a real workload.

The controller-runtime fake client is not a replacement for the API server. Use EnvTest when the test depends on validation, defaulting, subresources, watches, cache wiring, Generation, or ResourceVersion.

devenv test
devenv tasks run operator:bootstrap
devenv tasks run operator:doctor
devenv tasks run operator:ci-core

Run specialized lanes

• Use make fuzz for parsers, renderers, auth, or normalization code. Set FUZZTIME or FUZZ_TARGET_FILTER for a focused run.
• Use make verify-perf for changes that may affect reconcile cost, convergence, startup, or lifecycle timing.
• Use make test-e2e-existing for OpenShift and other platform-specific checks that Kind cannot reproduce.
• Use make test-e2e-operator-upgrade when controllers, CRDs, Helm, profiles, or migrations must preserve resources made by the prior stable operator.
• Use the manual HSM lane for KMIP and PKCS#11 paths that require test-only provider fixtures.

E2E suites are declared in test/e2e/suites.yaml. Update the suite owner, risk tier, isolation class, labels, coverage tags, CI lane, nightly policy, and parallelism whenever an E2E spec changes ownership or scope. Then run make verify-e2e-manifest.

Give each E2E scenario one unique case:<id> label. Keep the ID when the title, file, or lane changes. Declare the owning lanes in the suite’s ci.lanes and make their full selectors select each automatic scenario once. Each automatic owner must be included in the full CI matrix and use an automatic PR scope. The planner rejects missing IDs, undeclared or overlapping owners, and automatic scenarios without an executable owner. Manual platform and external-cluster scenarios retain their separate qualification requirements.

Generated matrices list expected_case_ids for every lane. CI validates those IDs against the Ginkgo report and requires every planned scenario to pass exactly once. A dry run can check selection but cannot prove execution.

Set E2E_FAIL_ON_EMPTY=true with label filters. Write Ginkgo JSON and JUnit reports so a failed lane retains selected specs, failure details, and slow-test evidence.

make test-e2e-ci \
  E2E_LABEL_FILTER='lifecycle && !openshift' \
  E2E_JUNIT_REPORT=artifacts/e2e-reports/local/junit.xml \
  E2E_JSON_REPORT=artifacts/e2e-reports/local/ginkgo.json \
  E2E_FAIL_ON_EMPTY=true

To reuse a compiled suite locally, build it from a clean checkout with the pinned Go toolchain. The default bundle location is artifacts/e2e-binaries. It records the current commit and platform; a bundle from another commit or platform is rejected.

make build-e2e-binaries
make test-e2e-ci \
  E2E_BINARY_DIR=artifacts/e2e-binaries \
  E2E_LABEL_FILTER='lifecycle && !openshift' \
  E2E_PARALLEL_NODES=2 \
  E2E_JUNIT_REPORT=artifacts/e2e-reports/local/junit.xml \
  E2E_JSON_REPORT=artifacts/e2e-reports/local/ginkgo.json \
  E2E_FAIL_ON_EMPTY=true

Add E2E_DRY_RUN=true to check selection in serial mode without running suite hooks or deleting a Kind cluster. Dry-run reports cannot qualify a lane. Without E2E_BINARY_DIR, make test-e2e-ci compiles the source suite as before. Run make test-ci-tools to check shared execution, bundle verification, worker coordination, and report validation without a cluster.

Continue with CI routing.

OpenBao configuration compatibility

Run make verify-openbao-config-compat with Docker available. The version matrix and fixture expectations live in hack/ci/openbao-config-compat/manifest.json. Each selected release uses its own upstream parser module and SDK. The default matrix uses 2.6.4 for migration compatibility and 2.7.1 as the current release. OpenBao 2.4 and 2.5 are outside this default gate. The check calls both ParseConfig and Validate, verifies parsed block counts and selected effective values, and rejects paths absent from the upstream parser or Raft backend inventory.

New configuration output needs a golden fixture and a manifest entry. The check fails if a golden file is unlisted or a declared operator configuration path has no fixture. Every golden fixture must have a positive case in the default matrix. Set version bounds and expected diagnostic fragments for version-specific features. The contract also checks removed built-in seals for a matching KMS plugin declaration. A negative case passes only when it detects the declared problem. An unrelated failure or an unexpectedly accepted configuration fails the check.

make verify-openbao-config-compat
make report-openbao-config-schema-drift
# Check one release while retaining the complete fixture contract.
bash hack/ci/openbao-config-compat.sh 2.7.1

Each run writes a new directory under dist/test/openbao-config-compat/ and prints its path. Set COMPAT_REPORT_DIR to select another output directory. Reports retain the upstream commit, version role, parser diagnostics, and contract problems separately. Nightly reports compare every selected release using qualified parser and profile paths, Raft configuration map accesses, and retry_join JSON fields. Provider-defined seal options and audit/API payloads remain outside that source inventory. Upstream options that the operator does not expose are informational, not compatibility failures.

This gate proves configuration recognition and the declared parsed-value assertions. It does not qualify provider credentials, plugin execution, API request effects, TLS negotiation, or certificate reload behavior. Use make test-tls-key-exchange-openbao for the released-server Raft and listener TLS contract. Use the focused TLS, initialization, and unseal E2E suites for their lifecycle and provider contracts.

Diagnose performance findings

Inspect the scenario artifacts before changing a performance baseline. Warmup scenario errors remain failures in the summary, even though warmup measurements do not enter the performance comparison. A scenario with no measured samples cannot pass.

The rolling-upgrade scenario writes an availabilityProbes JSONL artifact. Each probe records its timestamp, duration, HTTP status or transport error, and availability result. Failed HTTP responses retain up to 4 KiB of health or proxy response data. Failed probes also capture Pod readiness, deletion state, and public Service EndpointSlices. Errors while collecting this state remain separate from the probe result. The zero-failure availability gate remains in effect.

OpenBao Services select server Pods with openbao.org/component=openbaocluster. Helper Job Pods use other component values and do not serve OpenBao endpoints.

The restore scenario owns its source cluster and snapshot fixture. After observing a successful submission receipt, the harness acknowledges administrator Resume and waits for managed Pod replacement and hold release. restore_total_seconds measures the interval from the restore request to recovery release. The restore can remain Unknown: this measurement does not confirm snapshot application. The scenario records submission acceptance, recovery acknowledgement, and recovery release as separate phases.

Policy approval and repair

Run make test-policy-reconciliation-openbao with Docker available. The test starts a disposable OpenBao dev server on loopback and exercises the production policy client and reconciliation manager. It checks exact-content authorization, rejected extra parameters and legacy paths, deleted-policy repair, unchanged-policy writes, switching upgrade strategies under one approval, independent backup readiness, and retry cooldown after revocation. The Unit Tests CI job runs this check against OpenBao 2.6.4 and 2.7.1. The local command defaults to 2.6.4. Set POLICY_TEST_OPENBAO_IMAGE to test another OpenBao image.

Run make build-policy-approvals to generate both release approval files in dist/. The generator uses the same policy definitions as bootstrap and runtime reconciliation. For a specific manifest, run go run ./hack/tools/operator_policy_approval --cluster cluster.yaml. Operators use the release assets instead of building these tools.

Controller JWT isolation

Run make test-controller-jwt-openbao with Docker available. The test starts an EnvTest API server and two disposable OpenBao servers on loopback. It verifies Pod-bound token issuance, the controller’s ServiceAccount permission boundary, mode migration, and rejected cross-target and shared JWT replay with both inline and standard authentication. The Envtest Integration CI job runs this check against OpenBao 2.6.4 and 2.7.1. The local command defaults to 2.6.4; set OPENBAO_JWT_TEST_IMAGE to select another image. The test removes its containers when it finishes.

The controller-jwt E2E scenario runs the deployed controller against self-initialized OpenBao Pods. It verifies the generated controller role, Raft Autopilot repair, and approved policy repair after the original ten-minute JWT expires, with the same controller Pod and process. It also migrates an initialized Shared cluster, restores its pre-migration Raft snapshot, and repairs the restored audience restrictions through an independent administrator login.

make test-e2e-ci \
  E2E_LABEL_FILTER=controller-jwt \
  E2E_TIMEOUT=35m \
  E2E_FAIL_ON_EMPTY=true \
  E2E_JUNIT_REPORT=dist/test/controller-jwt/junit.xml \
  E2E_JSON_REPORT=dist/test/controller-jwt/ginkgo.json

This serial scenario uses the default inline transport and waits beyond the TokenRequest’s recorded expiration, with a minimum elapsed lifetime of ten minutes. It then requires OpenBao to reject that credential within 90 seconds, retaining the generated role’s clock-skew and expiration leeway. Periodic policy repair overlaps the lifetime wait. After rejection, an approved idle strategy change triggers policy verification without resetting credential identity, changing the OpenBao version, or backdating status. Allow up to 25 minutes for setup, reconciliation, and recovery. The snapshot operation uses the Raft API on the same CR; object storage, the OpenBaoRestore Job, and recovery into a different CR UID require their own E2E coverage.

The restore-minimal suite has two independently owned, ordered fixture groups. With E2E_PARALLEL_NODES=2, the real ten-minute identity-mismatch deadline runs alongside the other restore journeys. Each group owns its source and destination namespaces, Transit service, storage emulator, and credentials. The emulator uses a separate namespace because it runs as root; source and destination namespaces retain restricted Pod Security. Approval withdrawal and scheduled-source changes cannot affect the other group. Run make test-e2e-support to check fixture ownership and request-copy contracts without a cluster. When using test-e2e-existing with an isolated test cluster, set both E2E_OPENBAO_VERSION and E2E_OPENBAO_IMAGE, and configure its API-server egress addresses through E2E_API_SERVER_CIDR and E2E_API_SERVER_ENDPOINT_IPS.

The upgrade-rolling and upgrade-bluegreen-core lanes use two Ginkgo workers. Each ordered scenario owns its tenant namespace. Rolling snapshot recovery and native Blue/Green snapshot promotion also own separate RustFS namespaces; their endpoints and snapshot egress policies select that backend. Cleanup deletes storage with a namespace UID precondition and preserves evidence if tenant or storage cleanup fails. The scenarios retain their separate initial states, version pairs, replica counts, availability checks, and verification durations. Operation-lock scenarios retain Serial, Ordered because they pause the shared controller. Ginkgo runs them after parallel scenarios finish. Gateway API scenarios use a separate serial suite because they install shared CRDs and GatewayClasses. The verification, recovery, and Gateway lanes retain their existing worker counts.