cert-manager¶

TLS certificate automation via Let's Encrypt, using DNS-01 challenges through AWS Route53.

DNS-01 rather than HTTP-01 for one decisive reason: these hostnames resolve to RFC1918 addresses that Let's Encrypt cannot reach. Nothing on the public internet can complete an HTTP challenge against 10.9.2.248. Proving control of the DNS zone works from anywhere, and it is also the only way to get a wildcard.

At a glance¶


Namespace	`cert-manager`; the certificates it issues land in `kube-system`
Sync wave	`-5`; the ClusterIssuers at `2` and the Certificates at `3`
Depends on	External Secrets for the Route53 credential, so transitively on OpenBao
If it is down	Nothing immediately. Certificates stop renewing, and the consequence surfaces up to sixty days later
Health check	`kubectl get certificate -A` → all `READY=True`

Components¶

ClusterIssuers: Both staging (testing) and production issuers using DNS-01 via Route53.
Certificates: Wildcard TLS certs for *.k8s.wlkr.ch and *.infra.k8s.wlkr.ch, stored as Secrets in kube-system and referenced by the Gateways. Two certificates cover every hostname this cluster will ever serve, which is a pleasant place to be.

Sync order¶

The Application's own resources go in three sync waves, because each one cannot work until the one before it exists:

Wave	Resource	Needs
`1`	`ExternalSecret` `route53-credentials`	OpenBao, through ESO
`2`	`letsencrypt-staging`, `letsencrypt-prod`	The `route53-credentials` Secret
`3`	The three `Certificate`s	A `Ready` ClusterIssuer

Left in one wave, ArgoCD orders custom resources alphabetically — Certificate, then ClusterIssuer, then ExternalSecret, exactly backwards. The sync then waits on certificates that cannot issue until two resources behind them in the queue are applied. An issuer applied alongside the ExternalSecret fares no better: it comes up Ready=False with InvalidSolver ("failed to get secret route53-credentials") and stays there until something resyncs it.

The Application also sets a sync retry. Without one, a failed apply ends the operation where it fell and nothing picks it up again — and a single flake at the front of the chain, such as the external-secrets admission webhook being unreachable on a cluster whose CNI has only just come up, leaves every issuer and certificate behind it unmade.

Resources¶

Requests and limits are sized at roughly 2.5x the measured peak working set: controller 84Mi, cainjector 89Mi, webhook 24Mi. CPU is requested but not limited, like the rest of the platform.

The chart renders its ServiceMonitor unconditionally, so it cannot sync until the Prometheus operator CRDs exist. make install-cilium installs them during bootstrap; kube-prometheus-stack, which owns them, arrives several sync waves later.

AWS Credentials Setup¶

The DNS-01 solver needs AWS credentials with Route53 permissions. The route53-credentials Secret is materialised from OpenBao via an ExternalSecret.

Store the credentials in OpenBao once OpenBao and ESO are up:

bao kv put kv/cert-manager/route53 \
  access-key-id="YOUR_AWS_ACCESS_KEY_ID" \
  secret-access-key="YOUR_AWS_SECRET_ACCESS_KEY"

ESO will then create the route53-credentials Secret in the cert-manager namespace within refreshInterval (1h by default) — or, if you would rather not spend an hour wondering whether it worked, immediately:

kubectl annotate externalsecret -n cert-manager route53-credentials \
  force-sync=$(date +%s) --overwrite

The IAM user needs at minimum:

{
  "Effect": "Allow",
  "Action": ["route53:GetChange", "route53:ChangeResourceRecordSets", "route53:ListHostedZonesByName"],
  "Resource": "*"
}

Note

Until OpenBao is initialised, unsealed, and the secret is stored, cert-manager will fail to issue certificates. This is the dependency that catches people after every power cut: sealed OpenBao means no Route53 credentials, which means no renewals, which means an expired certificate roughly two months later with no obvious connection to the outage that caused it. For the very first bootstrap, see the Quickstart which walks through the order.

cert-manager is installed twice, sort of

make install-cert-manager — which make install-core calls — installs the chart and the ClusterIssuers by Helm before ArgoCD exists, and ArgoCD then adopts them. As with Cilium, there is only one pin: the Makefile reads targetRevision out of application.yaml rather than keeping a version of its own. On a single tainted node this is the target that hangs; see Single-node clusters.

Issuers¶

Issuer	Purpose
`letsencrypt-staging`	Testing — issues untrusted certs, no rate limits
`letsencrypt-prod`	Production — issues trusted certs, subject to rate limits

Use letsencrypt-staging first when setting up. Production allows five duplicate certificates per week, a misconfigured solver will retry cheerfully until that is gone, and then you wait — there is no appeals process and no amount of restarting the pod helps. Staging exists exactly so you can get it wrong as many times as you need to.

When a certificate will not issue¶

Work down the chain of custody; the answer is nearly always further back than the Certificate itself:

kubectl describe certificate -n kube-system <name>
kubectl get certificaterequest,order,challenge -A
kubectl -n cert-manager logs deploy/cert-manager --tail=100

A Challenge stuck in pending is a DNS problem, not a cert-manager problem: either the credentials cannot write to the zone, or the TXT record is there and the resolver has not caught up yet. dig +short TXT _acme-challenge.<host> settles which.

Directory Structure¶

cert-manager/                  # TLS Certificate Management
├── application.yaml           # ArgoCD Application (Helm chart)
├── values.yaml                # Helm values, shared with `make install-core`
├── cluster-issuers.yaml       # Let's Encrypt staging + prod issuers
├── certificates.yaml          # All Certificate resources
└── route53-credentials.yaml   # ExternalSecret → OpenBao