Skip to contentSkip to conversation

rtnetlink: Support fine-grained netdevice bulk deletion

PATCH Superseded Lahav Schlesinger 2 participants 3 messages Nov 25, 2021

Subsystems (2)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 rtnetlink: Support fine-grained netdevice bulk deletion

Under large scale, some routers are required to support tens of thousands of devices at once, both physical and virtual (e.g. loopbacks, tunnels, vrfs, etc).
At times such routers are required to delete massive amounts of devices at once, such as when a factory reset is performed on the router (causing a deletion of all devices), or when a configuration is restored after an upgrade, or as a request from an operator.

Currently there are 2 means of deleting devices using Netlink:
1. Deleting a single device (either by ifindex using ifinfomsg::ifi_index, or by name using IFLA_IFNAME)
2. Delete all device that belong to a group (using IFLA_GROUP)

Deletion of devices one-by-one has poor performance on large scale of devices compared to "group deletion":
After all device are handled, netdev_run_todo() is called which calls rcu_barrier() to finish any outstanding RCU callbacks that were registered during the deletion of the device, then wait until the refcount of all the devices is 0, then perform final cleanups.

However, calling rcu_barrier() is a very costly operation, each call taking in the order of 10s of milliseconds.

When deleting a large number of device one-by-one, rcu_barrier() will be called for each device being deleted.
As an example, following benchmark deletes 10K loopback devices, all of which are UP and with only IPv6 LLA being configured:

1. Deleting one-by-one using 1 thread : 243 seconds
2. Deleting one-by-one using 10 thread: 70 seconds
3. Deleting one-by-one using 50 thread: 54 seconds
4. Deleting all using "group deletion": 30 seconds

Note that even though the deletion logic takes place under the rtnl lock, since the call to rcu_barrier() is outside the lock we gain some improvements.

But, while "group deletion" is the fastest, it is not suited for deleting large number of arbitrary devices which are unknown a head of time. Furthermore, moving large number of devices to a group is also a costly operation.

This patch adds support for passing an arbitrary list of ifindex of devices to delete with a new IFLA_IFINDEX_LIST attribute. This gives a more fine-grained control over which devices to delete, while still resulting in rcu_barrier() being called only once. Indeed, the timings of using this new API to delete 10K devices is the same as using the existing "group" deletion.

The size constraints on the attribute means the API can delete at most 16382 devices in a single request.

---
v1 -> v2
- Unset 'len' of IFLA_IFINDEX_LIST in policy.
- Use __dev_get_by_index() instead of n^2 loop.
- Return -ENODEV if any ifindex is not present.
- Saved devices in an array.
- Fix formatting.

 include/uapi/linux/if_link.h |  1 +
 net/core/rtnetlink.c         | 53 ++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
include/uapi/linux/if_link.h +1 -0
index eebd3894fe89..f950bf6ed025 100644
@@ -348,6 +348,7 @@ enum {
 	IFLA_PARENT_DEV_NAME,
 	IFLA_PARENT_DEV_BUS_NAME,
 
+	IFLA_IFINDEX_LIST,
 	__IFLA_MAX
 };
 
net/core/rtnetlink.c +53 -0

71 lines not on this page · raw message

Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 1 line · On Thu, 25 Nov 2021 14:47:13 +0200 Lahav Schlesinger wrote:

+static int rtnl_list_dellink(struct net *net, void *ifindex_list, int size)

s/ifindex_list/ifindices/ ? it's not really a list

Can we make it an int pointer so we don't have to cast it later?

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3061

Quoted 7 lines
net/core/rtnetlink.c +7 -0
+{
+	const int num_devices = size / sizeof(int);
+	struct net_device **dev_list;
+	LIST_HEAD(list_kill);
+	int i, ret;
+
+	if (size < 0 || size % sizeof(int))

Does core reject size == 0? It won't be valid either.

Quoted 16 lines
net/core/rtnetlink.c +16 -0
+		return -EINVAL;
+
+	dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+	if (!dev_list)
+		return -ENOMEM;
+
+	for (i = 0; i < num_devices; i++) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;
+
+		ret = -ENODEV;
+		dev = __dev_get_by_index(net, ((int *)ifindex_list)[i]);
+		if (!dev)
+			goto out_free;
+
+		ret = -EOPNOTSUPP;

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3087

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+		ops = dev->rtnl_link_ops;
+		if (!ops || !ops->dellink)
+			goto out_free;
+
+		dev_list[i] = dev;
+	}
+
+	for (i = 0; i < num_devices; i++) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;

the temp variables are unnecessary here, the whole thing comfortably fits on a line:

		dev->rtnl_link_ops->dellink(dev_list[i], &list_kill);
Quoted 13 lines
net/core/rtnetlink.c +13 -0
+		dev = dev_list[i];
+		ops = dev->rtnl_link_ops;
+		ops->dellink(dev, &list_kill);
+	}
+
+	unregister_netdevice_many(&list_kill);
+
+	ret = 0;
+
+out_free:
+	kfree(dev_list);
+	return ret;
+}
#2 lore raw
Quoted 52 lines · Jakub Kicinski

CAUTION: External E-Mail - Use caution with links and attachments

Quoted 1 line · On Thu, 25 Nov 2021 14:47:13 +0200 Lahav Schlesinger wrote:

+static int rtnl_list_dellink(struct net *net, void *ifindex_list, int size)

s/ifindex_list/ifindices/ ? it's not really a list

Can we make it an int pointer so we don't have to cast it later?

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3061

Quoted 7 lines
net/core/rtnetlink.c +7 -0
+{
+     const int num_devices = size / sizeof(int);
+     struct net_device **dev_list;
+     LIST_HEAD(list_kill);
+     int i, ret;
+
+     if (size < 0 || size % sizeof(int))

Does core reject size == 0? It won't be valid either.

Quoted 16 lines
net/core/rtnetlink.c +16 -0
+             return -EINVAL;
+
+     dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+     if (!dev_list)
+             return -ENOMEM;
+
+     for (i = 0; i < num_devices; i++) {
+             const struct rtnl_link_ops *ops;
+             struct net_device *dev;
+
+             ret = -ENODEV;
+             dev = __dev_get_by_index(net, ((int *)ifindex_list)[i]);
+             if (!dev)
+                     goto out_free;
+
+             ret = -EOPNOTSUPP;

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3087

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+             ops = dev->rtnl_link_ops;
+             if (!ops || !ops->dellink)
+                     goto out_free;
+
+             dev_list[i] = dev;
+     }
+
+     for (i = 0; i < num_devices; i++) {
+             const struct rtnl_link_ops *ops;
+             struct net_device *dev;

the temp variables are unnecessary here, the whole thing comfortably fits on a line:

                dev->rtnl_link_ops->dellink(dev_list[i], &list_kill);
Quoted 13 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3101

Quoted 13 lines
net/core/rtnetlink.c +13 -0
+             dev = dev_list[i];
+             ops = dev->rtnl_link_ops;
+             ops->dellink(dev, &list_kill);
+     }
+
+     unregister_netdevice_many(&list_kill);
+
+     ret = 0;
+
+out_free:
+     kfree(dev_list);
+     return ret;
+}

Thanks for the feedback Jakub, I'll send a V3 with your proposed changes.