External Secrets Operator¶

External Secrets Operator (ESO) bridges the cluster's secret store (OpenBao) to native Kubernetes Secret objects. Workloads consume secrets the standard way (envFrom, volumeMounts, imagePullSecrets) without ever talking to OpenBao directly.

That last part is the whole trick. No sidecar, no init container, no application code that knows what a Vault token is. Applications keep reading environment variables like it is 2014, and the interesting work happens somewhere they never have to think about.

At a glance¶


Namespace	`external-secrets`
Sync wave	`-6`, the very first thing after the Gateway API CRDs — see why
Depends on	OpenBao at runtime, though not to be installed
If it is down	Secrets already materialised keep working. Nothing rotates, and nothing new resolves
Health check	`kubectl get clustersecretstore openbao` → `Valid`

Components¶

Resource	Scope	Purpose
`external-secrets`	Deployment	Reconciler that watches ExternalSecret objects
`ClusterSecretStore`	Cluster	Single store named `openbao` shared by all namespaces
`ExternalSecret`	Namespaced	One per Secret you want materialised in a namespace

How it works¶

sequenceDiagram
    participant App as Workload
    participant ES as ExternalSecret
    participant ESO as ESO Controller
    participant Bao as OpenBao
    participant K as K8s Secret

    ESO->>Bao: login (k8s SA JWT)
    Bao-->>ESO: vault token
    ESO->>Bao: read kv/path
    Bao-->>ESO: secret data
    ESO->>K: create/update Secret
    App->>K: mount / envFrom

ESO authenticates to OpenBao with its own ServiceAccount (external-secrets-vault in the external-secrets namespace). OpenBao validates the JWT against the Kubernetes TokenReview API and issues a short-lived OpenBao token bound to the external-secrets role/policy.

To get that JWT, ESO mints a short-lived token for the ServiceAccount through the TokenRequest API, which is what the external-secrets-vault-token-creator ClusterRole grants. The store uses ESO's vault provider unchanged — OpenBao implements the Vault HTTP API — and talks to the in-cluster Service, so no request leaves the cluster network.

Note what is absent from that sentence: any long-lived credential stored anywhere. The cluster's own identity system vouches for ESO, and OpenBao decides whether to believe it. This is the bootstrap problem solved properly, and it is worth understanding once rather than treating as magic — because when it breaks, it breaks in the ClusterSecretStore status and nowhere else.

Adding a new secret¶

The full round-trip from "I have a new credential" to "my Pod can read it":

1. Store the value in OpenBao¶

bao kv put kv/<workload>/<purpose> \
  key1="value1" \
  key2="value2"

Pick a path that follows the layout in OpenBao → Layout convention.

2. Declare an `ExternalSecret` in the workload's directory¶

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-credentials
  namespace: my-app
  annotations:
    # ESO and its CRDs install at sync-wave -6, ahead of everything that
    # consumes them. This annotation is what covers the first sync of a
    # cluster where they have not landed yet.
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: openbao
    kind: ClusterSecretStore
  target:
    name: my-app-credentials   # name of the resulting K8s Secret
    creationPolicy: Owner
  data:
    - secretKey: key1           # key in the K8s Secret
      remoteRef:
        key: <workload>/<purpose>
        property: key1          # key inside the KV entry
    - secretKey: key2
      remoteRef:
        key: <workload>/<purpose>
        property: key2

Drop the file alongside the workload's other manifests (deployment.yaml, service.yaml, …). The parent ArgoCD Application picks it up automatically.

3. Consume the Secret¶

spec:
  containers:
    - name: my-app
      envFrom:
        - secretRef:
            name: my-app-credentials

The Secret name in target.name is what shows up in the namespace.

Pulling whole secrets¶

To pull every key from a KV entry without listing them individually:

spec:
  dataFrom:
    - extract:
        key: <workload>/<purpose>

Refresh interval¶

refreshInterval: 1h polls OpenBao hourly. Set to 0 to disable polling — ESO will only re-sync on resource changes. For rotated credentials, leave it at a value that matches your rotation cadence.

Worth knowing: rotating a value in OpenBao updates the Kubernetes Secret within that interval, but it does not restart anything. A pod that read the secret into an environment variable at startup will happily keep using the old value until something restarts it. Reloader-style tooling or a rollout is the missing half of "rotation", and forgetting it is how a credential gets rotated on paper and not in practice.

Chart values¶

Sync wave -6. Ahead of every Application that ships an ExternalSecret; a CRD that does not exist yet deadlocks the wave rather than delaying it — see GitOps.
Webhook and cert-controller resources. Memory limits are about 2.5x the measured peak working set: 33Mi for the webhook, 73Mi for the cert-controller.
Webhook ServiceAccount token stays mounted. The webhook has no RoleBinding, so it looks as if it never talks to the API server, but it builds an in-cluster client at startup regardless and exits without the token (unable to load in-cluster config). It only shows when the pod is recreated.

Troubleshooting¶

`ClusterSecretStore` is not Ready¶

Start here for basically every secret-related problem. If the store is not Ready, nothing downstream of it will be either, and chasing the individual ExternalSecret first is wasted time.

kubectl get clustersecretstore openbao -o jsonpath='{.status}' | jq

Common causes:

OpenBao is sealed (bao status → Sealed: true) — unseal it, see OpenBao → Unsealing.
Kubernetes auth role missing — re-run the bao write auth/kubernetes/role/external-secrets … command.
Wrong serviceAccountRef — must point to external-secrets-vault in the external-secrets namespace.

An `ExternalSecret` is not Synced¶

kubectl describe externalsecret -n <namespace> <name>