Control Plane VIP¶

The Kubernetes API server is reached through a virtual IP held by kube-vip, not through one named node. Whichever control-plane node wins the leader election answers on control_plane_vip from ansible/inventory.yaml; if it goes away, another takes the address over.

This removes the dependency on a single control-plane node for cluster access — see Known Limitations for what still applies to a cluster whose certificates predate the VIP.

The alternative, which almost every first cluster does, is to name one node in the kubeconfig and quietly promote it to Most Important Machine. It works perfectly until that machine needs a reboot, at which point you discover that "highly available control plane" meant "three copies of etcd behind one hostname".

How it is wired¶

kube-vip runs in ARP mode with leader election, advertising a /32. It is deliberately configured with svc_enable: "false" — LoadBalancer services belong to Cilium's L2 announcements, and two components fighting over who gets to answer an ARP request produces the kind of intermittent, host-dependent weirdness that eats an entire evening.

The manifest is placed by Ignition rather than applied afterwards because it has to be running before kubeadm init writes the endpoint into the cluster's certificates. Chicken, egg, static pod.

Adoption by ArgoCD¶

A static pod only changes when someone rewrites the file on the node, which in practice means reinstalling it. So the static pod is only the bootstrap: once ArgoCD syncs payload/platform/kube-vip/, a DaemonSet on the control-plane nodes takes over, and upgrading kube-vip is a merged PR like any other chart.

On each node, the DaemonSet's adopt init container deletes /etc/kubernetes/manifests/kube-vip.yaml and waits for the static kube-vip to release :2112 before the new one starts. Both use the node name as their leader-election identity and the same plndr-cp-lock Lease, so the two must never run side by side on one node. Where the file is already gone, the init container does nothing.

A few details keep that handover short and safe:

• A pull init container runs first and only makes the kubelet fetch the kube-vip image, so the gap between the old process exiting and the new one starting does not include a registry round trip. Keep its image in step with the main container.
• adopt mounts /etc/kubernetes/manifests as a directory, because a hostPath mounted as a single file is a bind mount and cannot be removed. It gives up after 120 seconds of :2112 still being bound (the kubelet rescans manifests every 20 seconds), leaving a visible stuck Init rather than a second kube-vip beside something else holding the port.
• The rollout uses maxUnavailable: 1 with no surge, for the same reason: an old and a new pod would fight over :2112 and the lease identity.
• The pod tolerates every taint, since a NotReady or pressured control-plane node is exactly when the VIP has to move.
• Its capabilities are identical to the static pod's rather than tightened. The first sync replaces all three static pods at once, so an untested hardening there takes the VIP down everywhere.

A DaemonSet is usually the wrong home for the VIP, because a kubelet that reaches the API through the VIP cannot fetch the pod that would bring it up. That does not apply here: the control-plane kubelets use their own node's API server (check server: in /etc/kubernetes/kubelet.conf), so they still start kube-vip after a full power loss. kube-vip itself talks to the node's API server too, not to the kubernetes Service, which would need Cilium first: the DaemonSet sets KUBERNETES_SERVICE_HOST to the host IP (which is in the API server certificate's SANs), and container env wins over the variables the kubelet injects.

What that changes:

• kube_vip_version in the inventory is the bootstrap version only. Bump the image in daemonset.yaml to upgrade a running cluster.
• The first sync replaces all three static pods at once, so the VIP drops for a few seconds. Later updates roll one node at a time.
• The VIP settings live in two places, the inventory and daemonset.yaml. Keep them in step.
• kube-vip no longer uses admin.conf. Its kube-vip ServiceAccount may only get and update its own Lease.
• The Application does not prune, and its objects carry Delete=false: removing them takes the API away from the workers and from every kubeconfig.

Choosing the address¶

It must be a free address on the nodes' subnet, outside any DHCP range, and distinct from the Cilium LoadBalancer pools (10.9.2.248 and 10.9.2.249). Nothing validates this — a collision shows up as an unreachable API server after provisioning, or worse, as an API server that works from some machines and not others depending on whose ARP cache won.

Migrating a cluster built without a VIP¶

A cluster provisioned before this change has its first control-plane node's address baked into the API server certificates, so this is not a config change you can simply sync. Two options:

Rebuild (simplest)¶

Reprovision from the Quickstart. The VIP is in place from kubeadm init onwards and no migration is needed. Restore state per Backups & Recovery.

In place¶

Only worth it if rebuilding is not an option. Work on one node at a time and keep a second terminal open with a working kubeconfig — not as a nicety, but because several of the steps below can leave you unable to open a new one.

1. Confirm the address is free:

   ping -c2 <vip>        # must not answer
   arping -c2 <vip>      # from a node on the segment
   

2. Add the VIP to the API server certificate SANs. Edit the ClusterConfiguration stored in the cluster and add it under apiServer.certSANs:

   kubectl -n kube-system edit configmap kubeadm-config
   

3. Regenerate the API server certificate on each control-plane node:

   ssh core@<node>
   sudo mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.bak
   sudo mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.bak
   sudo kubeadm init phase certs apiserver
   sudo crictl ps | grep kube-apiserver   # confirm it restarted
   

4. Deploy the kube-vip static pod to each control-plane node. Generate it from the same values the template uses, or copy /etc/kubernetes/manifests/kube-vip.yaml from a node reprovisioned with the new configuration. Verify the VIP answers:

   curl -k https://<vip>:6443/healthz
   

5. Repoint the cluster at it. Update controlPlaneEndpoint in the kubeadm-config ConfigMap, then on every node rewrite the server address in /etc/kubernetes/*.conf and /var/lib/kubelet/kubeconfig and restart the kubelet.

6. Only now switch Cilium. Change k8sServiceHost in payload/platform/cilium/values.yaml to the VIP, commit, and let ArgoCD sync. Restart the Cilium DaemonSet and confirm every pod reconnects:

   kubectl -n kube-system rollout restart ds/cilium
   kubectl -n kube-system rollout status ds/cilium
   

7. Re-fetch your kubeconfig so it uses the VIP:

   make kubeconfig
   

Verifying¶

# which node currently holds the VIP
kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}'; echo
kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip -o wide

# the endpoint your kubeconfig actually uses
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo

To test failover, reboot the leader per Rebooting a node and confirm kubectl keeps working after a few seconds. Do this once, deliberately, on a quiet afternoon. Untested failover is not failover; it is a hypothesis.