Kubernetes Node Draining: A Guide to Safely Evicting Pods

Kubernetes Node Draining: A Guide to Safely Evicting Pods.

User Story: Say you need to update the spec of a running node that contains running pods, what will be your approach?

When a node needs to be taken offline for any reason, draining ensures that the running pods on that node are rescheduled to other available nodes in the cluster.

What is Node Drain?

Draining a node in Kubernetes means safely evicting all the running pods on that node

Draining Node following steps:

1. Cordon (marking it unschedulable)

2. Drain (safely evicting pods)

3. Monitoring the process.

Cordon: Cordon is a command used to mark a node as unschedulable, meaning no new pods will be scheduled on that node. Existing pods running on the node will continue to run, but the node will not accept any additional pods until it is uncordoned.


glab@k8smaster:~/salman$ kubectl cordon k8sworker2
node/k8sworker2 cordoned

k8sworker2 is SchedulingDisabled


glab@k8smaster:~/salman$ kubectl get nodes -n up-node
NAME                  STATUS                     ROLES         AGE  VERSION
k8smaster.example.net Ready                      control-plane 40d  v1.28.12
k8sworker1            Ready                      <none>        34d  v1.28.12
k8sworker2            Ready,SchedulingDisabled   <none>        2d2h v1.28.13
k8sworker3            Ready                      <none>        2d1h v1.28.13

Check pods are running or not, our all pods are running


glab@k8smaster:~/salman$ kubectl get pods -n up-node -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-5ccdc5f64c-c6nl2 1/1 Running 0 2m52s 172.16.230.247 k8sworker1 <none> <none>
nginx-deployment-5ccdc5f64c-mpx78 1/1 Running 0 2m52s 172.16.8.2 k8sworker2 <none> <none>
nginx-deployment-5ccdc5f64c-mtxrf 1/1 Running 0 2m52s 172.16.137.2 k8sworker3 <none> <none>

2. Drain: Drain is a command used to safely evict all running pods from a node in preparation for maintenance or decommissioning. Unlike cordon, which only prevents new pods from being scheduled on a node, drain actively removes all pods from the node and moves them to other available nodes in the cluster.




glab@k8smaster:~/salman$ kubectl drain k8sworker2 — ignore-daemonsets
node/k8sworker2  already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-fxjzh, kube-system/kube-proxy-5gqmf
evicting pod up-node/nginx-deployment-5ccdc5f64c-mpx78
evicting pod ingress-nginx/ingress-nginx-admission-patch-5h2zp
pod/ingress-nginx-admission-patch-5h2zp evicted
pod/nginx-deployment-5ccdc5f64c-mpx78 evicted
node/k8sworker2 drained

Our pod, which was running on the k8sworker2 node, has now been moved to run on another node, k8sworker3.


glab@k8smaster:~/salman$ kubectl get pods -n up-node -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-5ccdc5f64c-c6nl2 1/1 Running 0 7m13s 172.16.230.247 k8sworker1 <none> <none>
nginx-deployment-5ccdc5f64c-mtxrf 1/1 Running 0 7m13s 172.16.137.2 k8sworker3 <none> <none>
nginx-deployment-5ccdc5f64c-n69tl 1/1 Running 0 21s 172.16.137.3 k8sworker3 <none> <none>

Uncordon the node to mark it as schedulable again




glab@k8smaster:~/salman$ kubectl get nodes -n up-node
NAME STATUS ROLES AGE VERSION
k8smaster.example.net Ready control-plane 40d v1.28.12
k8sworker1 Ready <none> 34d v1.28.12
k8sworker2 Ready <none> 2d2h v1.28.13
k8sworker3 Ready <none> 2d1h v1.28.13

Node draining is a critical skill for Kubernetes administrators, ensuring that cluster maintenance happens without disrupting services. By mastering the kubectl drain command, you can confidently manage node outages, upgrades, and scaling operations. Have you used node draining in your Kubernetes environment?

Author

Md. Abu Salman

Software Engineer

Tirzok Private Limited

#1 Best Software Development Company In Bangladesh – Tirzok Private Limited