Troubleshooting¶

Start here, not with kubectl describe on the thing that looks broken. The thing that looks broken is usually downstream of something duller, and this section is ordered by how often the duller thing turns out to be the answer.

Where to look first¶

The table that saves the most time:

Two other symptom tables exist, for the phases this one does not cover:

• A node that will not install, or hangs at a blinking cursor: Troubleshooting PXE boot.
• A node that installs but comes up with no OSD: Rook-Ceph → No OSDs after reprovisioning.

Alerts are mailed by Alertmanager — see Monitoring → Alerting. The checks above are still worth running, because the delivery path itself is not monitored. An empty inbox means either that nothing is wrong or that the mail stopped working, and those two look identical from here.

Rolling back a bad sync¶

Everything under payload/ is applied by ArgoCD from Git, so the durable fix is a revert commit. To stop the bleeding first, disable auto-sync on the affected Application and roll it back:

kubectl -n argocd patch application <app> --type merge \
  -p '{"spec":{"syncPolicy":{"automated":null}}}'
argocd app rollback <app>

Re-enable auto-sync by restoring syncPolicy.automated once the revert has landed on main. Leaving it disabled means the Application silently stops tracking Git — and an Application that has quietly stopped tracking Git is a time bomb with a three-month fuse, defused only by somebody wondering why their change never took effect.