Backups & Recovery¶

Nobody wants backups. Everybody wants restores. This page is written with that distinction in mind: every mechanism below is described in terms of what it can actually get back for you, and — more importantly — what it cannot.

Current state¶

What is not covered¶

Alerting is the other half of a backup: a backup that silently stopped running is indistinguishable from one that works, right up until the moment you need it, which is also the moment you find out. Velero's PrometheusRule covers that, but only reaches whoever Alertmanager is configured to tell — see Monitoring for the state of that.

The bigger gap is where the backups land.

Every automated backup here goes into the same cluster's Ceph. That protects against the failures that actually happen — a deleted PVC, a bad prune, a corrupted database, a workload that ate its own data. It does not protect against losing the cluster, because the backups go with it.

This is worth sitting with for a moment rather than nodding past. A backup that shares a failure domain with its source covers operator error and nothing else. Operator error is genuinely the most common cause of data loss, so this is not worthless — it is just precisely one half of the job, and it is important to know which half you have.

Off-site replication is the missing piece. RGW supports bucket replication and Velero supports a second BackupStorageLocation, so the shape of the fix is known; neither is configured. Until then, treat the OpenBao unseal keys plus Git as the real disaster-recovery story and these backups as protection against mistakes rather than against the building burning down.

Velero¶

Velero backs up Kubernetes objects and volume data nightly at 02:00, keeping 14 days.

kube-system is excluded because it is reconstructed from Git on the next sync and is large. events are excluded because they expire anyway.

Why the data mover matters¶

A CSI snapshot on its own is a Ceph object. Backing up a PVC by snapshotting it would leave the only copy inside the same Ceph cluster the backup exists to survive — protection against a deleted PVC, but not against a broken pool. A snapshot is a bookmark, not a copy, and confusing the two is one of the more expensive mistakes available in storage. With defaultSnapshotMoveData, Velero takes the snapshot, streams the data out to the object store, and deletes the snapshot. The durable copy is the one in the bucket.

This is why deployNodeAgent is on: the node agent is what reads the snapshot and does the streaming.

Two settings that are not optional here¶

checksumAlgorithm: ""      # on the BackupStorageLocation config

The AWS plugin's aws-sdk-go-v2 sends a trailing checksum that Ceph RGW rejects with api error XAmzContentSHA256Mismatch, so every upload fails without this. The plugin's own README lists Ceph S3 as needing it. "S3-compatible" is one of the great load-bearing hyphens of our industry, and this is the sort of thing it is carrying.

image: velero/velero-plugin-for-aws:v1.14.2

Plugin v1.14.x pairs with Velero v1.18.x, this chart's appVersion. The chart's commented example still shows v1.13.1, which is the v1.17 line.

Other settings worth knowing¶

• No credentials file. credentials.useSecret is false: the AWS plugin reads its keys from the environment, and extraEnvVars feeds that straight from the Secret Rook writes for the velero-bucket claim. The keys are never rendered into a file or into Git.
• volumeSnapshotLocation: [] must stay empty. The chart ships a placeholder entry with a null name and provider, renders it as a VolumeSnapshotLocation called default, and the CRD schema rejects it, failing every sync. Helm replaces lists rather than merging them, so the empty list removes it. Nothing here needs one: CSI and the data mover use a VolumeSnapshotClass, and a VolumeSnapshotLocation belongs to the legacy per-provider snapshotter plugins.
• No runAsNonRoot. The plugin initContainer copies itself into /target, and whether that works as non-root depends on the image's own USER. Confirm it on a real backup before adding it; the seccomp profile, allowPrivilegeEscalation: false and dropped capabilities are set.
• Node agent sizing. Its limits come from a measured 37Mi peak. It is idle except during a backup, so the headroom is deliberately wide.

Snapshot plumbing¶

kubeadm does not install the CSI snapshot controller and neither does Rook. Without it the VolumeSnapshot CRDs are absent and the RBD driver advertises snapshot support nothing can invoke. snapshot-controller.yaml installs the controller and its CRDs at sync-wave 2, ahead of Velero at 3.

The VolumeSnapshotClass for rook-ceph-block is at wave 3 for the same reason, and that one is load-bearing. At the default wave it was applied before its CRD existed and failed with no matches for kind VolumeSnapshotClass. SkipDryRunOnMissingResource does not help there, since it is the apply that fails, not the dry run — and the failed sync meant the snapshot-controller and velero Applications queued behind it were never created at all.

Its deletionPolicy is Delete: the snapshot is only an intermediate step, and the durable copy is the one the data mover writes into the object store.

Using it¶

kubectl -n backup get backups.velero.io
kubectl -n backup get backupstoragelocation     # should be Available

# On demand
velero backup create manual-$(date +%s) --include-namespaces my-app

# Restore
velero restore create --from-backup velero-daily-20260905020000

If backups sit in PartiallyFailed, check that a VolumeSnapshotClass labelled velero.io/csi-volumesnapshot-class: "true" exists — without it Velero finds no class for the RBD driver and skips volumes silently, leaving you with a backup full of Kubernetes objects and none of the data anybody cared about.

Alerting¶

Velero ships a PrometheusRule here: VeleroBackupFailures (critical) and VeleroBackupPartialFailures (warning). A backup that silently stopped running is the failure mode this exists to prevent — and it is the one that catches experienced people, because the dashboard stays green and the CronJob still exists and everything looks exactly like it did last month.

etcd¶

Velero restores objects through the API server. That is the wrong tool for the case where there is no API server left to restore through — lost quorum, a corrupted data directory, three dead control-plane nodes. For that you need the etcd data itself, which Velero does not capture.

Two backup systems for two genuinely different disasters. It looks like redundancy until the day you need the one you skipped.

A CronJob takes one nightly at 01:00, an hour before Velero runs:

The snapshots go to their own bucket rather than Velero's because they are recovered by entirely different means. Two details of the job are easy to undo by accident:

• dnsPolicy: ClusterFirstWithHostNet — with hostNetwork the default policy uses the node's resolv.conf, which cannot resolve the RGW Service the upload needs.
• The upload writes an AWS CLI config file with addressing_style = path. RGW addresses buckets by path, the CLI defaults to virtual-host style, and there is no environment variable for it.

kubectl -n backup get cronjob etcd-backup
kubectl -n backup logs job/<most-recent-job> -c upload

Taking one by hand¶

Take a snapshot from a control-plane node:

kubectl -n kube-system exec -it etcd-<node> -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/snapshot.db

kubectl cp kube-system/etcd-<node>:/var/lib/etcd/snapshot.db ./etcd-snapshot.db

Copy it off the cluster. Now. A snapshot stored only on a Ceph PVC does not survive the failure it exists for, and "I'll move it later" has never once happened in the history of operations.

Restoring is etcdctl snapshot restore into a fresh data directory on a stopped control plane, then restarting kubelet — see the upstream kubeadm documentation.

OpenBao¶

kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot save /tmp/snapshot.bao
kubectl -n openbao cp openbao-0:/tmp/snapshot.bao ./openbao-snapshot.bao

The snapshot contains all KV data plus policies, roles and mounts. It does not contain the unseal keys, and it is useless without them — a perfect, verified, encrypted brick. Details in OpenBao → Backups.

Ceph volumes¶

Volume data is covered by Velero: PVCs are snapshotted nightly and the data is moved into the object store, so a deleted PVC is recoverable.

Ceph's own replication is not a backup and should not be mistaken for one. It spreads each block across OSDs, which protects against a disk or a node failing and against nothing else — not deletion, not corruption, not a bad prune. That last one is not hypothetical: every Application here runs with prune: true, so removing a PersistentVolumeClaim from Git deletes the volume.

What is still open is off-cluster replication — see What is not covered. RBD mirroring to a second cluster would close it.

Rebuilding from scratch¶

What you need, in order:

1. The Git repository — all platform and workload manifests.
2. The OpenBao unseal keys and root token — without these the restored OpenBao is an encrypted brick.
3. An OpenBao raft snapshot, or the willingness to re-enter every secret.
4. Optionally an etcd snapshot, to skip re-issuing certificates and waiting for the platform to re-converge.

Items 1 and 2 are the ones that actually matter, and only one of them lives somewhere GitHub keeps a copy. If you take a single action after reading this page, make it checking that you still know where those five key shares are.

The rebuild itself is the Quickstart from step 1. Because the platform is declarative, the cluster converges back to its documented state once ArgoCD points at the repository and OpenBao is unsealed.