A few days ago I was assigned a task to create pods with multiple network interfaces in the Kubernetes. It is not a difficult task, because it only requires a few annotations for the pods, provided that the CNI network has been correctly installed and configured. Multus and Danm are all such CNI networks, which supports multiple network interfaces. What takes me a lot of time in the task is the routing issue, when the pod wants to connect to a server. However, in this post I will not dive into the routing issue I encountered. Instead, I will briefly introduce the policy-based routing in Linux, which I learned during the issue debugging.
When a IP packet comes to a router, usually the packet is routed according to the destination IP address of the packet. However, a different algorithm is used in Linux. This is called policy-based routing, which uses not only the destination IP address of the packet, but also the source IP address of the packet, the incoming network interface of the packet, the source/destination port of the packet and so on to make the routing decision. In the following sections I will describe how to setup the policy based routing step by step.
1. create IP routing rules using the ‘ip route add SELECTOR ACTION’ command.
The SELECTOR should be something that selects a particular IP packet, for example, the source/destination IP address, the source/destination port, the incoming network interface and so on. The ACTION should be some action, which the routing system should do when the packet matches the SELECTOR criteria, For example, looking up some routing table, selecting some realm, and so on. In practice looking up some routing table is the most common action.
ip rule add from 10.10.10.10 table 100
ip rule add to 12.12.12.12 priority 300 table main
The first rule designates that the routes for the packets with source IP address 10.10.10.10 should be looked up in routing table with ID 100. The second rule designates that the routes for packets with destination IP address 12.12.12.12 should be looked up in routing table with ID 200. This IP routing rule has priority 300. The below command can be used to list the current IP routing rules:
#ip rule list
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
2. populate IP routing table using the ‘ip route add ROUTE’ command.
There are three routing tables by default in Linux, which are local, main, default tables. The local table is maintained by Linux kernel and usually shouldn’t be modified. The main table is setup by Linux kernel and can be modified. This table contains some routes that are populated by Linux during bootup. The default table usually is empty.
A custom routing table can be created by adding a number (1-255) and a name to /etc/iproute2/rt_tables file. For example,
echo 200 custom >> /etc/iproute2/rt_tables
This creates a routing table with ID 200 and name ‘custom’. With a routing table in hand, we can populate routes in the table now using ‘ip route add ROUTE’ command. For example,
ip route add table 200 to 192.168.10.0/24 dev eth1
ip route add table 200 to 192.168.20.0/24 via 10.10.10.23 dev eth0
ip route add table 200 default via 10.10.10.23 dev eth0
The first route specifies that the packets to subnet 192.168.10.0/24 should be routed to eth1 network interface, which connects to a switch. The packet will be directly routed to the destination by the switch. The second route specifies that the packets to subnet 192.168.20.0/24 should be routed to the gateway with IP 10.10.10.23 and the gateway can be reached directly through eth0 network interface. The third route specifies that all the other packets should be routed to the gateway with IP 10.10.10.23 through eth0 network interface. The gateway will know how to route those packets to the next hope.
This is a brief introduction about policy-based routing in Linux. Hope this can help you.