Security Policies¶

Three cluster-wide controls: Pod Security Admission, default-deny network policy, and no API token on the default ServiceAccount.

Both are the kind of thing that is easy to turn on and hard to turn on safely. The pattern used here — measure first, enforce second, one namespace at a time — is unglamorous and is the only approach that survives contact with a running cluster.

At a glance¶

Pod Security Admission¶

PSA is the built-in replacement for PodSecurityPolicy, and a namespace that does not opt in runs at the default privileged level — which enforces nothing at all. Every namespace you have not thought about is wide open, silently, by default. That is the fact this section exists to address.

Each namespace carries three labels, and the split between them is the point:

Enforcement is set to what already works, so nothing running breaks, while warn and audit show what a stricter level would have caught. Tightening enforcement later becomes an informed change rather than a guess — and a security control that broke production once is a security control that gets switched off permanently, which is the real risk here.

Namespace objects are owned by this Application so the labels stay declarative rather than being applied once by CreateNamespace=true and then drifting. Pruning is disabled for the whole Application — pruning a Namespace deletes everything inside it, including PVCs, so a misplaced deletion in Git becomes an irreversible data loss event roughly three minutes later. That is not a mistake worth leaving available to a sleepy Tuesday.

Checking what would break¶

kubectl label --dry-run=server --overwrite ns cert-manager \
  pod-security.kubernetes.io/enforce=restricted

That reports every pod in the namespace that would be rejected, without changing anything.

It only sees what is running right now, though — a CronJob that fires nightly and violates restricted is invisible to a dry run at three in the afternoon. The audit label covers that gap by recording violations continuously, and those records are queryable in Grafana over the full Loki retention window:

{job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations"

Use the dry run to check the common case and the audit log to catch the rest. Between them, tightening enforce stops being a guess.

Network policies¶

Without a policy, pod-to-pod traffic is unrestricted across all namespaces: anything with a foothold in one pod can reach OpenBao's API, the Ceph mons and the Kubernetes API alike. The flat network is Kubernetes' default and it is almost never what anyone actually wants — it is just what you get for free.

Why CiliumNetworkPolicy and not NetworkPolicy¶

This is the detail that makes or breaks a default-deny here. Two kinds of traffic have no pod identity a standard NetworkPolicy can name:

• Gateway traffic. With Cilium's Gateway API implementation, a request reaching a backend arrives from the per-node Envoy, not from a pod.
• Health probes. kubelet's liveness and readiness probes come from the node itself.

A plain NetworkPolicy default-deny therefore kills all ingress and all health probes — the pods then fail their probes and restart forever, which looks exactly like an application fault and not at all like a policy one. Expect to spend a while reading application logs before it occurs to you that the application is fine and the kubelet is the one being blocked. Cilium's fromEntities names these directly: ingress for Envoy-proxied traffic, host and remote-node for the kubelet.

Scope¶

Ingress only, in eight namespaces:

Every policy also admits traffic from within the namespace and from host and remote-node for probes. The monitoring rule for kured is easy to miss and costly to lose: kured queries Prometheus before every reboot and blocks when the query fails, so without it no node ever reboots.

Egress is deliberately untouched. A default-deny on egress also needs rules for DNS, the API server, and every external endpoint each component talks to; getting that wrong takes the component down rather than merely leaving it exposed. And you will get it wrong at least twice, because nobody has a complete list of what their components phone home to. It is the next step to take, not one that is taken here.

The rest are not covered and allow all ingress, each for its own reason:

Rolling this out safely¶

One namespace at a time, watching Hubble between each. Rolling out network policy everywhere at once is how people end up reverting the whole thing at midnight and never trying again:

kubectl -n kube-system port-forward svc/hubble-relay 4245:80
hubble observe --verdict DROPPED --namespace openbao --follow

If something legitimate is being dropped, Cilium can put a single endpoint into audit mode — policy decisions are logged but not enforced — which is the fastest way to find a missing rule without an outage:

kubectl -n kube-system exec ds/cilium -- \
  cilium endpoint config <endpoint-id> PolicyAuditMode=Enabled

Default ServiceAccount tokens¶

Every pod that does not opt out gets a projected API token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. For a workload that never calls the API server, that is a credential handed to a process with no use for it — and the first thing anything with code execution in the container picks up.

The default ServiceAccount is the worst case. Kubernetes creates one in every namespace, nothing configures it, and a pod that names no serviceAccountName silently lands on it. A workload reaching the API through default is already a misconfiguration, so default-serviceaccounts.yaml sets automountServiceAccountToken: false on it everywhere with nothing to preserve. That includes namespaces with no workloads at all (default, kube-public, kube-node-lease, gateway-system, cilium-secrets): a mountable token in an empty namespace stops being harmless the day something is deployed there.

Two namespaces are left out, because something really runs on their default SA:

• kube-system. etcd, the API server, controller-manager and scheduler run there. They authenticate with client certificates from kubeconfigs on the node, so removing the token would very likely be fine — and "very likely fine" is not the standard for the control plane.
• authentik. authentik-server sets no serviceAccountName, while authentik-worker beside it uses the authentik SA. That asymmetry is an upstream chart default, to be fixed at the source rather than worked around.

Directory Structure¶

security/
├── application.yaml              # ArgoCD Application, prune disabled
├── pod-security.yaml             # Namespace objects with PSA labels
├── network-policies.yaml         # Default-deny ingress, eight namespaces
└── default-serviceaccounts.yaml  # No token automount on `default` SAs