Operations¶
Everything in this section assumes a cluster that is already up. For first provisioning see the Quickstart.
Building a cluster is a weekend. Operating one is the rest of your life. This is the section you will actually come back to.
What is in this section¶
-
The symptom-to-first-check table, and how to roll back a sync that made things worse. Start here when something is wrong.
-
Rebooting, adding, replacing and rebuilding a node — including the two gates people skip: waiting for Ceph, and unsealing OpenBao afterwards.
-
The three mechanisms that update a node, why none of them finishes on its own, and how to move a minor version deliberately.
-
What is backed up, what is not, and the fact worth knowing before you need it: every automated backup lands inside the cluster it is backing up.
-
How the API server address survives losing the node that answers it, and how to migrate a cluster that was built without it.
Routine health check¶
Four commands, thirty seconds, run them when you walk past the rack. The second column is what you are actually looking for, because three of the four will happily print something reassuring while being wrong:
| Command | Healthy looks like |
|---|---|
kubectl get nodes |
Every node Ready |
kubectl -n argocd get applications |
Every Application Synced and Healthy |
kubectl get certificate -A |
Every certificate READY=True |
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status |
HEALTH_OK, and an OSD count matching your node count |
A fifth is worth adding after anything restarted, because it is the one failure that looks like health:
The Rook toolbox pod is enabled, so ceph status, ceph osd tree and
ceph health detail are available without installing anything.
Note
Ceph metrics are scraped — monitoring.enabled is true in the CephCluster spec, and createPrometheusRules ships Ceph's own alerting rules — so a degraded pool or a down OSD reaches Prometheus without anyone running ceph status. Run it anyway: it is the fastest way to see why, and it is what Kured is really asking about before it reboots anything. See Rook-Ceph → Monitoring.