openshift-node-operations

name: openshift-node-operations description: Node lifecycle management with focus on safe drain procedures, automated-vs-manual infrastructure decisions, and node failure diagnosis.

OpenShift Node Operations

Key Decision: Automated vs. Manual Infrastructure

This fork affects EVERY node operation. Determine it first.

• Automated infrastructure (AWS, Azure, GCP, OpenStack): Nodes are managed by MachineSets. You scale MachineSets, not nodes. The Machine controller handles provisioning, joining, and decommissioning. Never manually delete node objects — delete the Machine object instead.
• Manual infrastructure (bare metal, pre-provisioned VMs): You handle the full lifecycle — provisioning RHCOS, ignition config, CSR approval, labeling, and hardware decommission. The cluster only knows about nodes, not machines.

Safe Drain Procedure

1. Cordon first, then drain. If you drain without cordoning, new pods can schedule on the node during drain — you get a moving target that never finishes draining.
2. Check PDBs before draining. A PodDisruptionBudget with maxUnavailable: 0 or minAvailable equal to current replica count will block drain silently — it just hangs. Run oc get pdb -A and check if any PDB protects pods on your target node.
3. Use grace periods for stateful workloads. The default grace period may not be enough for apps that need to flush data or close connections. Set --grace-period explicitly for databases and message queues.
4. --force loses data. Force drain deletes pods with emptyDir volumes without waiting for graceful shutdown. Only use for node decommission, never for maintenance where you expect the node to return.
5. --delete-emptydir-data is required for most drains because system pods (metrics, logging) use emptyDir. Without this flag, drain refuses to proceed. This is safe — it's the user-data emptyDir volumes you need to worry about, and --force is what skips their graceful handling.

Adding Nodes

Automated Infrastructure

Scale the MachineSet: oc scale machineset <name> -n openshift-machine-api --replicas=<N>. Monitor the Machine status (not just node status) — a Machine stuck in "Provisioning" means the cloud provider call failed (quota, network, AMI issues).

Manual Infrastructure (Bare Metal)

After provisioning and booting with ignition, watch for TWO rounds of CSRs:

1. First CSR: the node-bootstrapper requests a client certificate to join the cluster
2. Second CSR: the node itself requests a serving certificate

Both must be approved. Don't bulk-approve with xargs in production without inspecting each CSR — a rogue CSR could grant access to an unauthorized node.

Removing and Replacing Nodes

Automated Infrastructure

Delete the Machine object, NOT the node object. The MachineSet controller notices the replica count is short and creates a replacement automatically. Deleting just the node object orphans the underlying VM — it keeps running and costing money but the cluster doesn't know about it.

Manual Infrastructure

Cordon → drain → oc delete node <name> → decommission the hardware/VM. The cluster does not manage the underlying infrastructure, so deleting the node object is the final cluster-side step.

Replacement gotcha

For automated infra, simply deleting the Machine triggers replacement. For manual infra, you must provision a new machine from scratch (RHCOS + ignition + CSR approval + labeling).

Node Failure Diagnosis Priority

When a node shows NotReady, check in this order (most common causes first):

1. Network: Can the node reach the API server? If the kubelet can't phone home, the node goes NotReady even though it's otherwise healthy. Check SDN/OVN pods on the node.
2. Disk pressure: /var/lib/containers/ fills up from accumulated images. Clean with crictl rmi --prune via oc debug node/. Also check journal size — journalctl --vacuum-time=3d reclaims space.
3. Memory pressure: Find the top consumers with oc adm top pods -A — it may be system workloads (monitoring, logging) not user pods causing pressure.
4. Kubelet not running: oc debug node/<name>, then chroot /host && systemctl status kubelet. Check journalctl -u kubelet for the actual error.
5. Certificate issues: Expired kubelet certs cause NotReady with no obvious symptoms in pod logs. Check CSRs with oc get csr — pending CSRs for the node indicate cert renewal problems.

Gotchas

• MachineSet edits only affect NEW machines. Changing instance type or labels in a MachineSet does not update existing machines. To apply changes, you must delete existing machines and let the MachineSet recreate them.
• Node labels applied manually are lost on replacement. If the Machine is deleted and recreated, the new node gets labels from the MachineSet template, not the old node. Always set persistent labels in MachineSet spec.template.spec.metadata.labels.
• oc debug node/ creates a privileged pod on the node. If the node can't schedule pods (disk full, kubelet down), debug won't work either. In that case, SSH is the only option.
• Deleting a node object does NOT deprovision the VM/machine. The cloud provider keeps billing for it. Always delete the Machine object for automated infrastructure.

When to Use Sibling Skills

• Machine API operator issues → use openshift-operator-troubleshooting
• Node stuck during cluster upgrade → use openshift-cluster-upgrade for MCP diagnosis
• Workloads not rescheduling after drain → use openshift-debugging for pod-level triage