net-next: When a bond have a massive amount of VLANs with IPv6 addresses, performance of changing link state, attaching a VRF, changing an IPv6 address, etc. go down dramtically.
Subsystems (2)
PATCH v3 1/1 net-next: When a bond have a massive amount of VLANs with IPv6 addresses, performance of changing link state, attaching a VRF, changing an IPv6 address, etc. go down dramtically.
The source of most of the slow down is the dev_addr_lists.c module, which mainatins a linked list of HW addresses.
When using IPv6, this list grows for each IPv6 address added on a VLAN, since each IPv6 address has a multicast HW address associated with it.
When performing any modification to the involved links, this list is traversed many times, often for nothing, all while holding the RTNL lock.
Instead, this patch adds an auxilliary rbtree which cuts down traversal time significantly.
Performance can be seen with the following script:
#!/bin/bash
ip netns del test || true 2>/dev/null
ip netns add test
echo 1 | ip netns exec test tee /proc/sys/net/ipv6/conf/all/keep_addr_on_down > /dev/null
set -e
ip -n test link add foo type veth peer name bar ip -n test link add b1 type bond
ip -n test link add florp type vrf table 10
ip -n test link set bar master b1
ip -n test link set foo up
ip -n test link set bar up
ip -n test link set b1 up
ip -n test link set florp up
VLAN_COUNT=1500
BASE_DEV=b1
echo Creating vlans
ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link add link $BASE_DEV name foo.\$i type vlan id \$i; done"
echo Bringing them up
ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link set foo.\$i up; done"
echo Assiging IPv6 Addresses
ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test address add dev foo.\$i 2000::\$i/64; done"
echo Attaching to VRF
ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link set foo.\$i master florp; done"
On an Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz machine, the performance before the patch is (truncated):
Creating vlans real 108.35 Bringing them up real 4.96 Assiging IPv6 Addresses real 19.22 Attaching to VRF real 458.84
After the patch:
Creating vlans real 5.59 Bringing them up real 5.07 Assiging IPv6 Addresses real 5.64 Attaching to VRF real 25.37
---
In version 1, Nikolay Aleksandrov <[email protected]> asked why not remove the list entirely in favour of the tree.
Altough this sounds like the right solution,
I am not sure it's possible to do this with RCU-safety in mind, and I doubt my ability to correctly patch every place in the kernel that uses the list without introducing further bugs.
v1 -> v2:
- Formatting/typo fixes
- Retarget net-next
v2 -> v3:
- Exclude first list address from tree because it can be changed via `dev->dev_addr` (Thanks <[email protected]>)
--- include/linux/netdevice.h | 5 ++ net/core/dev_addr_lists.c | 144 ++++++++++++++++++++++++++------------ 2 files changed, 103 insertions(+), 46 deletions(-)
include/linux/netdevice.h +5 -0
@@ -48,6 +48,7 @@#include <uapi/linux/if_bonding.h> #include <uapi/linux/pkt_cls.h> #include <linux/hashtable.h> +#include <linux/rbtree.h> struct netpoll_info ;struct device ;
@@ -202,6 +203,7 @@ struct sk_buff;struct netdev_hw_addr {struct list_head list; +struct rb_node node;unsigned char addr[MAX_ADDR_LEN ];unsigned char type;#define NETDEV_HW_ADDR_T_LAN 1
@@ -219,6 +221,9 @@ struct netdev_hw_addr {struct netdev_hw_addr_list {struct list_head list;int count; + +/* Auxiliary tree for faster lookup on addition and deletion */ +struct rb_root tree; };#define netdev_hw_addr_list_count (l) ((l)->count)
net/core/dev_addr_lists.c +98 -46
249 lines not on this page · raw message
Signature · 1 line
2.25.1
Jakub Kicinski
maintainerPlease repost with the subject fixed.
It should be something like:
[PATCH net-next v4] net: maintain rbtree of device hw addresses
not the first sentence of the commit message.