Kube-Proxy: Switching from iptables to IPVS mode

Kube-Proxy: Switching from iptables to IPVS mode

User Story: As we know service to pod traffic load balancing is random as iptables is used by kube-proxy as default mode of operation, is there any way so that we ensure to round robin

By default, Kubernetes uses iptables for handling network traffic.

To check if iptables is being used as the network proxy in your Kubernetes cluster.

1. Check Kube-Proxy Mode

Run the following command to see the current configuration of the kube-proxy:

kubectl get cm kube-proxy -n kube-system -o yaml

In the output, check the mode field under kubeProxyConfiguration:
If mode is set to iptables or is empty (""), kube-proxy is using iptables.
If mode is set to ipvs, then it’s using IPVS.

2. We can check another way to check network proxy

glab@k8smaster:~$ kubectl get pods -n kube-system -l k8s-app=kube-proxy
NAME               READY   STATUS    RESTARTS       AGE
kube-proxy-424vz   1/1     Running   43 (67m ago)   60d
kube-proxy-5gqmf   1/1     Running   20 (67m ago)   28d
kube-proxy-h8msk   1/1     Running   45 (67m ago)   66d
kube-proxy-xvjz8   1/1     Running   20 (66m ago)   27d

glab@k8smaster:~$ kubectl logs kube-proxy-424vz -n kube-system
I0930 04:17:02.761466 1 server_others.go:69] "Using iptables proxy"

To change from iptables to IPVS mode in a Kubernetes cluster for kube-proxy, follow these steps:

3. Verify IPVS Support
Make sure that your nodes support IPVS by checking the installed kernel modules:

lsmod | grep ip_vs

If you ran lsmod | grep ip_vs and got an empty result, it means the required IPVS kernel modules are not loaded on your system. To enable IPVS mode in Kubernetes, you need to install the IPVS-related kernel modules.

Here’s how to install and load the IPVS modules:

4. Install Required Packages

For Ubuntu/Debian:

sudo apt-get update
sudo apt-get install ipvsadm

5. Load IPVS Kernel Modules

After installing ipvsadm, load the necessary IPVS kernel modules.
Run the following commands to load the modules:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

These modules enable IPVS with different load-balancing algorithm:

ip_vs: Base IPVS module
ip_vs_rr: Round-robin algorithm
ip_vs_wrr: Weighted round-robin
ip_vs_sh: Source hash scheduling
nf_conntrack: Required for connection tracking

6. Make IPVS Modules Load at Boot

To ensure that the IPVS modules load automatically on system reboot, create a configuration file /etc/modules-load.d/ipvs.conf:

sudo tee /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

This will ensure that the modules are loaded every time the system starts.

7. Verify IPVS Modules Are Loaded

Run the following command again to confirm that the IPVS modules are now loaded:

lsmod | grep ip_vs

You should see output similar to this:

ip_vs_sh               16384 0
ip_vs_wrr              16384 0
ip_vs_rr               16384 0
ip_vs                 147456 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          139264 3 xt_conntrack,nf_nat,nf_conntrack_ipv4

8. Modify Kube-Proxy Configuration

Now, modify the kube-proxy configuration to use IPVS mode instead of iptables.

kubectl edit cm kube-proxy -n kube-system

Look for the mode field under kubeProxyConfiguration. Change the mode from iptables to ipvs like this:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"

9. Restart the Kube-Proxy Pods

After editing the configuration, restart the kube-proxy pods to apply the change

Kubernetes will recreate the kube-proxy pods automatically with the new IPVS mode.

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

10. Verify the Change

glab@k8smaster:~$ kubectl get pods -n kube-system -l k8s-app=kube-proxy
NAME               READY   STATUS    RESTARTS   AGE
kube-proxy-c9j6p   1/1     Running   0          41m
kube-proxy-dkhwv   1/1     Running   0          41m
kube-proxy-rs98t   1/1     Running   0          41m
kube-proxy-v87b4   1/1     Running   0          41m

To verify that IPVS is now being used, check the logs of a kube-proxy pod:

glab@k8smaster:~$ kubectl logs kube-proxy-c9j6p -n kube-system
I0930 05:39:52.847584       1 node.go:141] Successfully retrieved node IP: 192.168.122.198
I0930 05:39:52.849038       1 conntrack.go:52] "Setting nf_conntrack_max" nfConntrackMax=131072
I0930 05:39:52.870909       1 server.go:632] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
I0930 05:39:52.905574       1 server_others.go:218] "Using ipvs Proxier"

11. To convert from IPVS back to iptables mode in your Kubernetes cluster

Edit the Kube-Proxy ConfigMap
First, you'll need to modify the kube-proxy configuration to switch the mode from IPVS to iptables.

Run the following command to edit the kube-proxy ConfigMap:

kubectl edit cm kube-proxy -n kube-system

In the editor, look for the mode field under kubeProxyConfiguration. If it is currently set to ipvs, change it to iptables, or simply leave it empty ("") to default to iptables.

mode: iptables

Or:

mode: ""

12. Delete Existing Kube-Proxy Pods

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

13. Verify the Change

Check the logs of the newly created kube-proxy pods:

kubectl logs -n kube-system -l k8s-app=kube-proxy

Look for the following message to confirm iptables mode:

Using iptables Proxier.

Summary

To convert from IPVS back to iptables:

● Edit the kube-proxy ConfigMap and set the mode to iptables (or leave it empty).

● Delete the kube-proxy pods to apply the changes.

● Verify that kube-proxy is now using iptables by checking the logs and iptables rules.

This process ensures a smooth transition from IPVS back to iptables mode for service networking in your Kubernetes cluster.

Author
Md. Abu Salman
Software Engineer
Tirzok Private Limited
LinkedIn

Kubernetes Node Draining: A Guide to Safely Evicting Pods