OpenBao¶

OpenBao is the cluster's secret store — an open-source, Linux Foundation fork of HashiCorp Vault. It holds every secret consumed by workloads on the cluster (cloud credentials, API tokens, registry pulls, database passwords, …). Secrets are surfaced to Kubernetes as native Secret objects via the External Secrets Operator.

It is also the single most consequential component on this cluster. When OpenBao is unhappy, nothing that needs a credential works, and the failure presents as six unrelated things breaking at once. Learn its two states — sealed and unsealed — before you need to.

flowchart LR
    Operator([Operator]) -->|bao CLI| Bao[(OpenBao<br/>KV v2)]
    ESO[External Secrets<br/>Operator] -->|read| Bao
    ESO -->|create/update| KSecret[K8s Secret]
    App[App Pod] -->|env / volume| KSecret

    style Bao fill:#e1f5ff,stroke:#0288d1

At a glance¶

Architecture¶

The chart is the official upstream openbao/openbao-helm, pinned in application.yaml.

Chart values that are not what they look like¶

• Pod security context. OpenBao keeps the root key out of swap with mlock, which needs IPC_LOCK — the reason the namespace enforces privileged. The container still drops ALL and adds back only IPC_LOCK. The chart's securityContext.pod is an if/else, not a merge: setting it replaces the default block, so runAsUser: 100, runAsGroup: 1000 and fsGroup: 1000 (the chart's own defaults) are restated. Drop them and keep only runAsNonRoot, and the kubelet refuses to start the container, because the image's USER is the name openbao rather than a number and cannot be proven non-root. It only shows on pod recreation, typically after a node reboot.
• Image repository. The chart prepends server.image.registry, which defaults to quay.io; the repository is therefore openbao/openbao, not quay.io/openbao/openbao.
• Unauthenticated metrics. /v1/sys/metrics otherwise wants a token, and the ServiceMonitor has none, so the target would sit at 403. The metrics carry counts and timings, not paths or secrets, and are readable only by what can reach port 8200 — the namespace policy's callers plus the Gateway.
• Dashboard. serverTelemetry.grafanaDashboard renders OpenBao's upstream dashboard (grafana.com 23725) into the namespace.

Bootstrap¶

ArgoCD provisions the StatefulSet, PVCs, Services, and the vault.infra.k8s.wlkr.ch HTTPRoute. The pods go Ready within seconds, before the cluster is initialised or unsealed — see Ready does not mean unsealed. Neither happens on its own: initialisation is a one-time step, and unsealing is a step you will repeat after every restart.

Two commands do all of it:

make bao-init      # initialise, unseal, configure the engine and ESO's auth
make bao-secrets   # populate the five paths the cluster reads

The rest of this section is what those do, in the order they do it — worth reading once, because the failure modes are much easier to recognise if you know what was supposed to happen.

1. Initialise the cluster (one-time)¶

kubectl -n openbao exec -it openbao-0 -- bao operator init \
  -key-shares=5 \
  -key-threshold=3

The command prints 5 unseal keys and an initial root token. Store them in a password manager, right now, before you run another command. Not in the terminal scrollback. Not in a note you will "tidy up later". Losing all 5 keys means the data is unrecoverable, and OpenBao is not being dramatic about that — there is no support line, no recovery flow, and no clever trick. There is just the ciphertext and no way in.

2. Unseal each replica¶

The seal is Shamir, so nothing unseals these pods but you. Repeat for openbao-0, openbao-1, openbao-2, providing 3 of the 5 keys each time:

for pod in openbao-0 openbao-1 openbao-2; do
  for i in 1 2 3; do
    kubectl -n openbao exec -it "$pod" -- bao operator unseal
  done
done

Three of five, three times, once per pod. Yes, it is tedious — that tedium is the entire security model, and it is the price of keeping the key material off every machine but yours.

A pod that predates those stanzas, or that started before openbao-0 was initialised, needs pointing at the leader once. It takes unseal keys afterwards:

kubectl -n openbao exec openbao-1 -- \
  bao operator raft join http://openbao-0.openbao-internal:8200

Confirm the result with:

kubectl -n openbao exec -it openbao-0 -- bao status

You should see Initialized: true, Sealed: false, HA Mode: active on one pod and standby on the others.

3. Authenticate locally¶

For convenience, port-forward and point the CLI at the local instance:

kubectl -n openbao port-forward svc/openbao 8200:8200 &
export BAO_ADDR=http://127.0.0.1:8200
bao login   # paste the root token

The remaining steps assume bao is configured this way.

Secret engine¶

A single KV v2 engine is mounted at the path kv/. All cluster secrets live under it. One engine, one convention, no debates six months from now about whether it was kv/ or secret/.

bao secrets enable -path=kv -version=2 kv

Layout convention¶

kv/
├── authentik/
│   └── config             # secret-key, postgres-password, bootstrap-password,
│                          # bootstrap-token, and the client id/secret pairs
│                          # ArgoCD and Grafana read back from here
├── cert-manager/
│   └── route53            # access-key-id, secret-access-key
├── external-dns/
│   └── route53            # access-key-id, secret-access-key
├── monitoring/
│   ├── grafana-admin      # password
│   └── smtp               # username, password, to
└── <workload>/<purpose>   # one leaf per secret

Five paths, seven ExternalSecrets: authentik/config is read by three of them, because generating the OIDC client credentials up front is what keeps both sides of each integration declarative. The two route53 leaves are deliberately separate and meant to be separate IAM users -- cert-manager only writes _acme-challenge TXT records, while external-dns can repoint hostnames. Anything added later follows the same <workload>/<purpose> shape.

The bao kv put for each path lives in a comment at the top of the ExternalSecret that consumes it, which is the list to trust; they are collected in Quickstart step 11.

Each leaf is a single secret with one or more keys. ExternalSecret resources reference paths as cert-manager/route53 (the KV v2 data/ prefix is added by ESO automatically).

Storing a secret¶

bao kv put kv/cert-manager/route53 \
  access-key-id="AKIA..." \
  secret-access-key="..."

Avoiding it means writing the secret as JSON and handing bao the file, which is what make bao-secrets does. @ is parsed before anything is sent, so a missing file fails loudly rather than storing half a secret:

bao kv put -mount=kv cert-manager/route53 @/tmp/secret.json

Reading a secret¶

bao kv get kv/cert-manager/route53

Kubernetes auth method¶

External Secrets Operator authenticates to OpenBao using ServiceAccount JWTs. Set this up once after init:

# Enable the auth method
bao auth enable kubernetes

# Tell OpenBao how to reach the cluster's TokenReview API. The CA cert
# and host are read from the in-cluster ServiceAccount projection.
bao write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

The OpenBao ServiceAccount (openbao in namespace openbao) already has the system:auth-delegator ClusterRole bound to it via rbac.yaml in this directory, so the TokenReview calls succeed without additional setup.

Policy for ESO¶

bao policy write external-secrets - <<'EOF'
path "kv/data/*" {
  capabilities = ["read"]
}
path "kv/metadata/*" {
  capabilities = ["read", "list"]
}
EOF

Role binding ESO's ServiceAccount¶

bao write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets-vault \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets \
  ttl=1h

The external-secrets-vault ServiceAccount is created by payload/platform/external-secrets/cluster-secret-store.yaml — see External Secrets.

Once this is done, ExternalSecret resources cluster-wide will resolve. Verify with:

kubectl get externalsecret -A
kubectl get clustersecretstore openbao -o yaml

The Status.Conditions of the ClusterSecretStore should report Ready=True.

Unsealing after a restart¶

OpenBao seals itself on every pod restart — every node reboot, every ArgoCD upgrade, every chart bump, every time a kubelet has a bad day. This is by design and it is not going to stop:

make bao-unseal

That checks each replica and feeds three shares to whichever are sealed, reading them from output/credentials/openbao-init.json. It waits for a replica that has not joined the raft cluster yet rather than failing on it, and it is idempotent — on an unsealed cluster it says so and stops.

With the key file deleted, which is the correct end state, it is the loop below instead, three shares per pod:

for pod in openbao-0 openbao-1 openbao-2; do
  kubectl -n openbao exec "$pod" -- bao status >/dev/null 2>&1
  case $? in
    0) echo "$pod: already unsealed" ;;
    2) for i in 1 2 3; do
         kubectl -n openbao exec -it "$pod" -- bao operator unseal
       done ;;
    *) echo "$pod: OpenBao did not answer, check the pod" ;;
  esac
done

bao status exits 0 when unsealed, 2 when sealed and 1 when it cannot reach the server, and kubectl exec passes that exit code through.

Ready does not mean unsealed¶

The readiness probe calls /v1/sys/health with sealedcode=204 and uninitcode=204, so a sealed or uninitialised replica answers healthy and the pod goes Ready seconds after it starts. That is deliberate: if readiness waited for unsealing, a StatefulSet rolling update would stop after the first pod until someone typed in the keys, and every chart bump and Kured reboot would hang half-applied.

The cost is that kubectl get pods cannot tell you whether OpenBao is usable. Only bao status can, and it is what make bao-unseal and the loop above check.

Nothing does this for you. There is no auto-unseal seal configured, so a reboot at 03:00 leaves the cluster running and its secret store shut until someone with the key shares logs in. Plan for that rather than being surprised by it: while OpenBao is sealed no ExternalSecret resolves, so cert-manager loses the Route53 credentials it needs to renew certificates.

The failure is slow, which is what makes it dangerous. Nothing breaks the day OpenBao seals; things break sixty days later when a certificate expires and nobody connects the two events. See OpenBao needs an operator to unseal it.

Backups¶

The Raft storage backend supports snapshotting:

bao operator raft snapshot save snapshot.bao

Snapshots include all KV data and OpenBao's own config (policies, roles, mounts). Store them off-cluster — a snapshot on a PVC inside the cluster it is meant to rebuild is decoration. Restore with bao operator raft snapshot restore. And note the obvious: the snapshot is encrypted with a key that exists only in those five shares, so it is exactly as recoverable as your key custody is.

Directory Structure¶

openbao/
├── application.yaml                # ArgoCD Application (Helm: openbao/openbao)
├── httproute.yaml                  # vault.infra.k8s.wlkr.ch
└── rbac.yaml                       # system:auth-delegator binding for the openbao SA