Audit Logging¶

The API server records who did what, to which object, and whether it was allowed. This page covers what is recorded, the policy that decides it, where the log ends up, and how to read it.

At a glance¶

Until this was configured the API server recorded none of it: --audit-log-path was unset, so no audit log existed at all.

That absence had a second consequence that is easy to miss. Every namespace in pod-security.yaml carries an audit label set to a stricter level than it enforces — and the destination for an audit finding is the API server audit log. Without one, half of that design was writing to nowhere. The warn half still reached whoever ran kubectl apply; the record nobody was watching in real time, which is the half that matters afterwards, did not exist.

The policy decides everything¶

--audit-log-path on its own does nothing. Without --audit-policy-file the API server declines to open a log, and the failure is silent — no error, no file, an audit configuration that looks present in the manifest and produces nothing. The policy lives in ansible/templates/butane_node_config.yaml.j2 and is written by Ignition to /etc/kubernetes/audit/policy.yaml.

Rules are evaluated top to bottom and the first match wins, so the order is the design:

Where the log actually lives¶

The audit log is written to the root filesystem, which on an installed node is 125 GB of ext4 rather than the tmpfs the nodes used to run from. That is what makes the CIS rotation numbers affordable:

The filesystem is written once, on the install, and persists from then on — so the audit log now survives a reboot on its own. Alloy still tails it into Loki, and that is still where the copy that matters lives: an audit log stored only on the node is unavailable in precisely the situation where the node is what failed, and unqueryable next to everything else in the meantime.

Reading it¶

Audit events reach Loki as JSON with job="kubernetes-audit", plus verb and audit_level as labels. Everything else stays in the line, where | json can reach it:

# Who read Secrets, and which ones
{job="kubernetes-audit"} | json | objectRef_resource="secrets"

# Every exec into a running container
{job="kubernetes-audit", verb="create"} | json | objectRef_subresource="exec"

# Pod Security Admission violations that were audited but not enforced
{job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations"

That last query is the one the PSA labels were always meant to feed. It answers "what would break if I tightened enforce on this namespace" from evidence rather than from a dry run, and it answers it for the whole retention window rather than for the moment you happened to look.

Applying it to a running cluster¶

Changing the templates changes what a newly provisioned node gets. A running control-plane node keeps the kube-apiserver.yaml static pod manifest that kubeadm rendered at init time, and nothing rewrites it on its own.

# 1. Regenerate the Ignition configs so a rebuild gets this too
make config

# 2. Place the policy file
ssh core@<node> sudo mkdir -p /etc/kubernetes/audit /var/log/kubernetes/audit
ssh core@<node> sudo chmod 0700 /etc/kubernetes/audit /var/log/kubernetes/audit
scp output/.../policy.yaml core@<node>:/tmp/policy.yaml
ssh core@<node> sudo install -m 0600 /tmp/policy.yaml /etc/kubernetes/audit/policy.yaml

# 3. Re-render the API server static pod from the updated config
ssh core@<node> sudo kubeadm init phase control-plane apiserver \
  --config /opt/kubeadm-config.yaml

# 4. The kubelet restarts the static pod within seconds
kubectl -n kube-system get pod kube-apiserver-<node> -w

Then update the kubeadm-config ConfigMap in kube-system so the flags survive the next kubeadm upgrade or control-plane join — the ConfigMap is what kubeadm reads on those paths, not /opt/kubeadm-config.yaml.