Skip to contentSkip to conversation

rtnetlink: Support fine-grained netdevice bulk deletion

PATCH Superseded Lahav Schlesinger 5 participants 18 messages Dec 5, 2021 – Jan 4, 2022

Subsystems (2)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 rtnetlink: Support fine-grained netdevice bulk deletion

Under large scale, some routers are required to support tens of thousands of devices at once, both physical and virtual (e.g. loopbacks, tunnels, vrfs, etc).
At times such routers are required to delete massive amounts of devices at once, such as when a factory reset is performed on the router (causing a deletion of all devices), or when a configuration is restored after an upgrade, or as a request from an operator.

Currently there are 2 means of deleting devices using Netlink:
1. Deleting a single device (either by ifindex using ifinfomsg::ifi_index, or by name using IFLA_IFNAME)
2. Delete all device that belong to a group (using IFLA_GROUP)

Deletion of devices one-by-one has poor performance on large scale of devices compared to "group deletion":
After all device are handled, netdev_run_todo() is called which calls rcu_barrier() to finish any outstanding RCU callbacks that were registered during the deletion of the device, then wait until the refcount of all the devices is 0, then perform final cleanups.

However, calling rcu_barrier() is a very costly operation, each call taking in the order of 10s of milliseconds.

When deleting a large number of device one-by-one, rcu_barrier() will be called for each device being deleted.
As an example, following benchmark deletes 10K loopback devices, all of which are UP and with only IPv6 LLA being configured:

1. Deleting one-by-one using 1 thread : 243 seconds
2. Deleting one-by-one using 10 thread: 70 seconds
3. Deleting one-by-one using 50 thread: 54 seconds
4. Deleting all using "group deletion": 30 seconds

Note that even though the deletion logic takes place under the rtnl lock, since the call to rcu_barrier() is outside the lock we gain some improvements.

But, while "group deletion" is the fastest, it is not suited for deleting large number of arbitrary devices which are unknown a head of time. Furthermore, moving large number of devices to a group is also a costly operation.

This patch adds support for passing an arbitrary list of ifindex of devices to delete with a new IFLA_IFINDEX attribute. A single message may contain multiple instances of this attribute).
This gives a more fine-grained control over which devices to delete, while still resulting in rcu_barrier() being called only once. Indeed, the timings of using this new API to delete 10K devices is the same as using the existing "group" deletion.

---
v4 -> v5
- Call ->dellink() only once for duplicated devices.

v3 -> v4
- Change single IFLA_INDEX_LIST into multiple IFLA_IFINDEX
- Fail if passing both IFLA_GROUP and at least one IFLA_IFNEX

v2 -> v3
- Rename 'ifindex_list' to 'ifindices', and pass it as int*
- Clamp 'ops' variable in second loop.

v1 -> v2
- Unset 'len' of IFLA_IFINDEX_LIST in policy.
- Use __dev_get_by_index() instead of n^2 loop.
- Return -ENODEV if any ifindex is not present.
- Saved devices in an array.
- Fix formatting.

 include/uapi/linux/if_link.h |  1 +
 net/core/rtnetlink.c         | 82 ++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
include/uapi/linux/if_link.h +1 -0
index eebd3894fe89..68fcde9c0c5e 100644
@@ -348,6 +348,7 @@ enum {
 	IFLA_PARENT_DEV_NAME,
 	IFLA_PARENT_DEV_BUS_NAME,
 
+	IFLA_IFINDEX,
 	__IFLA_MAX
 };
 
net/core/rtnetlink.c +82 -0

112 lines not on this page · raw message

Signature · 1 line
2.25.1
#1 lore raw

On 05/12/2021 11:36, Lahav Schlesinger wrote:
[snip]

Quoted 13 lines

But, while "group deletion" is the fastest, it is not suited for deleting large number of arbitrary devices which are unknown a head of time. Furthermore, moving large number of devices to a group is also a costly operation.

This patch adds support for passing an arbitrary list of ifindex of devices to delete with a new IFLA_IFINDEX attribute. A single message may contain multiple instances of this attribute).
This gives a more fine-grained control over which devices to delete, while still resulting in rcu_barrier() being called only once. Indeed, the timings of using this new API to delete 10K devices is the same as using the existing "group" deletion.

Quoted 2 lines

---

[snip]

Note that Ido raised a good point[1] about group operations and their efficiency as a reply to v4 of the patch.

+CC Ido

[1] https://marc.info/?l=linux-netdev&m=163869796129581&w=2

#2 lore raw
In reply to Lahav Schlesinger

Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.

[snip]

include/uapi/linux/if_link.h enum { · line 351

Quoted 9 lines
include/uapi/linux/if_link.h +1 -0
index eebd3894fe89..68fcde9c0c5e 100644
@@ -348,6 +348,7 @@ enum {
 	IFLA_PARENT_DEV_NAME,
 	IFLA_PARENT_DEV_BUS_NAME,
 
+	IFLA_IFINDEX,

nit: maybe the previous blank line sit better after this new attribute (and before __IFLA_MAX)?

Quoted 16 lines
include/uapi/linux/if_link.h +0 -0
 	__IFLA_MAX
 };
 
net/core/rtnetlink.c +1 -0
index fd030e02f16d..5165cc699d97 100644
@@ -37,6 +37,7 @@
 #include <linux/pci.h>
 #include <linux/etherdevice.h>
 #include <linux/bpf.h>
+#include <linux/sort.h>
 
 #include <linux/uaccess.h>
 

net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884

Quoted 4 lines
net/core/rtnetlink.c +1 -0
@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
 	[IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED },
 	[IFLA_NEW_IFINDEX]	= NLA_POLICY_MIN(NLA_S32, 1),
 	[IFLA_PARENT_DEV_NAME]	= { .type = NLA_NUL_STRING },
+	[IFLA_IFINDEX]		= { .type = NLA_S32 },

Same policy than IFLA_NEW_IFINDEX to refuse negative ifindex.

Quoted 35 lines
net/core/rtnetlink.c +28 -0
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
 	return 0;
 }
 
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+	struct net_device * const *dev1 = a, * const *dev2 = b;
+
+	return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+				struct netlink_ext_ack *extack)
+{
+	int i = 0, num_devices = 0, rem;
+	struct net_device **dev_list;
+	const struct nlattr *nla;
+	LIST_HEAD(list_kill);
+	int ret;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		if (nla_type(nla) == IFLA_IFINDEX)
+			num_devices++;
+	}
+
+	dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+	if (!dev_list)
+		return -ENOMEM;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+		int ifindex;
+
+		if (nla_type(nla) != IFLA_IFINDEX)
+			continue;
+
+		ifindex = nla_get_s32(nla);
+		ret = -ENODEV;
+		dev = __dev_get_by_index(net, ifindex);
+		if (!dev) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");

It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099

Quoted 7 lines
net/core/rtnetlink.c +7 -0
+			goto out_free;
+		}
+
+		ret = -EOPNOTSUPP;
+		ops = dev->rtnl_link_ops;
+		if (!ops || !ops->dellink) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Device cannot be deleted");

Same here.

Thank you,
Nicolas

#3 lore raw
Quoted 36 lines · Nicolas Dichtel
Quoted 36 lines
net/core/rtnetlink.c +32 -0
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
 	return 0;
 }
 
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+	struct net_device * const *dev1 = a, * const *dev2 = b;
+
+	return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+				struct netlink_ext_ack *extack)
+{
+	int i = 0, num_devices = 0, rem;
+	struct net_device **dev_list;
+	const struct nlattr *nla;
+	LIST_HEAD(list_kill);
+	int ret;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		if (nla_type(nla) == IFLA_IFINDEX)
+			num_devices++;
+	}
+
+	dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+	if (!dev_list)
+		return -ENOMEM;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;
+		int ifindex;
+
+		if (nla_type(nla) != IFLA_IFINDEX)
+			continue;
Quoted 8 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092

Quoted 6 lines
net/core/rtnetlink.c +6 -0
+
+		ifindex = nla_get_s32(nla);
+		ret = -ENODEV;
+		dev = __dev_get_by_index(net, ifindex);
+		if (!dev) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");

It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".

extack infra does not allow dynamic messages. It does point to the location of the bad index, so userspace could figure it out.

Quoted 14 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099

Quoted 8 lines
net/core/rtnetlink.c +7 -0
+			goto out_free;
+		}
+
+		ret = -EOPNOTSUPP;
+		ops = dev->rtnl_link_ops;
+		if (!ops || !ops->dellink) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Device cannot be deleted");

Same here.

Thank you,
Nicolas

#7 lore raw
In reply to Lahav Schlesinger
Quoted 13 lines · Lahav Schlesinger
net/core/rtnetlink.c +1 -0
index fd030e02f16d..5165cc699d97 100644
@@ -37,6 +37,7 @@
 #include <linux/pci.h>
 #include <linux/etherdevice.h>
 #include <linux/bpf.h>
+#include <linux/sort.h>
 
 #include <linux/uaccess.h>
 

net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884

Quoted 4 lines
net/core/rtnetlink.c +1 -0
@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
 	[IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED },
 	[IFLA_NEW_IFINDEX]	= NLA_POLICY_MIN(NLA_S32, 1),
 	[IFLA_PARENT_DEV_NAME]	= { .type = NLA_NUL_STRING },
+	[IFLA_IFINDEX]		= { .type = NLA_S32 },

you need to make sure this new attribute can not be used in setlink requests or getlink.

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3057

Quoted 10 lines
net/core/rtnetlink.c +3 -0
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
 	return 0;
 }
 
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+	struct net_device * const *dev1 = a, * const *dev2 = b;

const struct net_device *dev1 = 1, *dev2 = b;

Quoted 7 lines
net/core/rtnetlink.c +7 -0
+
+	return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+				struct netlink_ext_ack *extack)
+{

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3074

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+	int i = 0, num_devices = 0, rem;
+	struct net_device **dev_list;
+	const struct nlattr *nla;
+	LIST_HEAD(list_kill);
+	int ret;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		if (nla_type(nla) == IFLA_IFINDEX)
+			num_devices++;
+	}

The need to walk the list twice (3 really with the sort) to means the array solution is better.

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3086

Quoted 12 lines
net/core/rtnetlink.c +12 -0
+
+	dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+	if (!dev_list)
+		return -ENOMEM;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;
+		int ifindex;
+
+		if (nla_type(nla) != IFLA_IFINDEX)
+			continue;

and this business too. We have arrays in other places (net/ipv4/nexthop.c), so this is not the first.

Quoted 11 lines
net/core/rtnetlink.c +11 -0
+
+		ifindex = nla_get_s32(nla);
+		ret = -ENODEV;
+		dev = __dev_get_by_index(net, ifindex);
+		if (!dev) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");
+			goto out_free;
+		}
+
+		ret = -EOPNOTSUPP;
+		ops = dev->rtnl_link_ops;

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3107

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+		if (!ops || !ops->dellink) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Device cannot be deleted");
+			goto out_free;
+		}
+
+		dev_list[i++] = dev;
+	}
+
+	/* Sort devices, so we could skip duplicates */
+	sort(dev_list, num_devices, sizeof(*dev_list), dev_ifindex_cmp, NULL);

how did this sort change the results? 10k compares and re-order has to add some overhead.

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3112

Quoted 5 lines
net/core/rtnetlink.c +5 -0
+
+	for (i = 0; i < num_devices; i++) {
+		struct net_device *dev = dev_list[i];
+
+		if (i != 0 && dev_list[i - 1]->ifindex == dev->ifindex)
		if (i && ...)

I liked the array variant better. Jakub?

#4 lore raw
In reply to David Ahern

Le 06/12/2021 à 16:20, David Ahern a écrit :

Quoted 46 lines · Nicolas Dichtel
Quoted 36 lines
Quoted 36 lines
net/core/rtnetlink.c +32 -0
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
 	return 0;
 }
 
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+	struct net_device * const *dev1 = a, * const *dev2 = b;
+
+	return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+				struct netlink_ext_ack *extack)
+{
+	int i = 0, num_devices = 0, rem;
+	struct net_device **dev_list;
+	const struct nlattr *nla;
+	LIST_HEAD(list_kill);
+	int ret;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		if (nla_type(nla) == IFLA_IFINDEX)
+			num_devices++;
+	}
+
+	dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+	if (!dev_list)
+		return -ENOMEM;
+
+	nla_for_each_attr(nla, head, len, rem) {
+		const struct rtnl_link_ops *ops;
+		struct net_device *dev;
+		int ifindex;
+
+		if (nla_type(nla) != IFLA_IFINDEX)
+			continue;
Quoted 8 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092

Quoted 6 lines
net/core/rtnetlink.c +6 -0
+
+		ifindex = nla_get_s32(nla);
+		ret = -ENODEV;
+		dev = __dev_get_by_index(net, ifindex);
+		if (!dev) {
+			NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");

It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".

Quoted 2 lines

extack infra does not allow dynamic messages. It does point to the location of the bad index, so userspace could figure it out.

Oh yes, my bad.

Thank you,
Nicolas

#5 lore raw
In reply to Nicolas Dichtel
Quoted 17 lines · Nicolas Dichtel

CAUTION: External E-Mail - Use caution with links and attachments

Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.

[snip]

Quoted 8 lines
include/uapi/linux/if_link.h +0 -0
index eebd3894fe89..68fcde9c0c5e 100644
@@ -348,6 +348,7 @@ enum {
      IFLA_PARENT_DEV_NAME,
      IFLA_PARENT_DEV_BUS_NAME,
 
Quoted 3 lines

include/uapi/linux/if_link.h enum { · line 351

Quoted 1 line
include/uapi/linux/if_link.h +1 -0
+     IFLA_IFINDEX,

nit: maybe the previous blank line sit better after this new attribute (and before __IFLA_MAX)?

Due to the comment above the previous 2 attributes, I think that by removing this empty line it can be accidentally thought as if the new attribute is part of this "block".
As for adding a new line before __IFLA_MAX, I wanted to preserve the appearance we had before the IFLA_PARENT_DEV_xxx attributes where added, where there was no empty line before __IFLA_MAX.

I don't mind either way though, whatever looks better to you.

Quoted 14 lines
Quoted 14 lines
include/uapi/linux/if_link.h +0 -0
      __IFLA_MAX
 };
 
net/core/rtnetlink.c +1 -0
index fd030e02f16d..5165cc699d97 100644
@@ -37,6 +37,7 @@
 #include <linux/pci.h>
 #include <linux/etherdevice.h>
 #include <linux/bpf.h>
+#include <linux/sort.h>
 
Quoted 8 lines

net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884

Quoted 7 lines
net/core/rtnetlink.c +1 -0
 #include <linux/uaccess.h>
 
@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
      [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED },
      [IFLA_NEW_IFINDEX]      = NLA_POLICY_MIN(NLA_S32, 1),
      [IFLA_PARENT_DEV_NAME]  = { .type = NLA_NUL_STRING },
+     [IFLA_IFINDEX]          = { .type = NLA_S32 },

Same policy than IFLA_NEW_IFINDEX to refuse negative ifindex.

Right, thanks

Quoted 61 lines
Quoted 36 lines
net/core/rtnetlink.c +28 -0
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
      return 0;
 }
 
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+     struct net_device * const *dev1 = a, * const *dev2 = b;
+
+     return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+                             struct netlink_ext_ack *extack)
+{
+     int i = 0, num_devices = 0, rem;
+     struct net_device **dev_list;
+     const struct nlattr *nla;
+     LIST_HEAD(list_kill);
+     int ret;
+
+     nla_for_each_attr(nla, head, len, rem) {
+             if (nla_type(nla) == IFLA_IFINDEX)
+                     num_devices++;
+     }
+
+     dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+     if (!dev_list)
+             return -ENOMEM;
+
+     nla_for_each_attr(nla, head, len, rem) {
+             const struct rtnl_link_ops *ops;
+             struct net_device *dev;

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+             int ifindex;
+
+             if (nla_type(nla) != IFLA_IFINDEX)
+                     continue;
+
+             ifindex = nla_get_s32(nla);
+             ret = -ENODEV;
+             dev = __dev_get_by_index(net, ifindex);
+             if (!dev) {
+                     NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");

It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099

Quoted 7 lines
net/core/rtnetlink.c +7 -0
+                     goto out_free;
+             }
+
+             ret = -EOPNOTSUPP;
+             ops = dev->rtnl_link_ops;
+             if (!ops || !ops->dellink) {
+                     NL_SET_ERR_MSG_ATTR(extack, nla, "Device cannot be deleted");

Same here.

Thank you,
Nicolas

maintainer
#8 lore raw
In reply to David Ahern
Quoted 2 lines · On Mon, 6 Dec 2021 21:12:33 -0700 David Ahern wrote:

The need to walk the list twice (3 really with the sort) to means the array solution is better.

Technically the linear walk will be dwarfed by the sort for non-trivial inputs (have I been conducting too many coding interviews?)

Especially with retpolines :(

Quoted 1 line

I liked the array variant better. Jakub?

I always default to attr-per-member unless there is a strong constraint on message size. Trying to extend old school netlink (TC etc) left me scarred. But we can do array, no big deal.

#6 lore raw
In reply to Lahav Schlesinger

Le 07/12/2021 à 13:48, Lahav Schlesinger a écrit :

Quoted 22 lines · Nicolas Dichtel
Quoted 17 lines

CAUTION: External E-Mail - Use caution with links and attachments

Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.

[snip]

Quoted 8 lines
include/uapi/linux/if_link.h +0 -0
index eebd3894fe89..68fcde9c0c5e 100644
@@ -348,6 +348,7 @@ enum {
      IFLA_PARENT_DEV_NAME,
      IFLA_PARENT_DEV_BUS_NAME,
 
Quoted 3 lines

include/uapi/linux/if_link.h enum { · line 351

Quoted 1 line
include/uapi/linux/if_link.h +1 -0
+     IFLA_IFINDEX,

nit: maybe the previous blank line sit better after this new attribute (and before __IFLA_MAX)?

Quoted 6 lines

Due to the comment above the previous 2 attributes, I think that by removing this empty line it can be accidentally thought as if the new attribute is part of this "block".
As for adding a new line before __IFLA_MAX, I wanted to preserve the appearance we had before the IFLA_PARENT_DEV_xxx attributes where added, where there was no empty line before __IFLA_MAX.

Good point.

Quoted 2 lines

I don't mind either way though, whatever looks better to you.

Ok, forget my comment.

Regards,
Nicolas

#10 lore raw
In reply to David Ahern
Quoted 16 lines · David Ahern

CAUTION: External E-Mail - Use caution with links and attachments

Quoted 12 lines · Lahav Schlesinger
net/core/rtnetlink.c +1 -0
index fd030e02f16d..5165cc699d97 100644
@@ -37,6 +37,7 @@
 #include <linux/pci.h>
 #include <linux/etherdevice.h>
 #include <linux/bpf.h>
+#include <linux/sort.h>
 
 #include <linux/uaccess.h>
 
Quoted 8 lines

net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884

Quoted 5 lines
net/core/rtnetlink.c +1 -0
@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
      [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED },
      [IFLA_NEW_IFINDEX]      = NLA_POLICY_MIN(NLA_S32, 1),
      [IFLA_PARENT_DEV_NAME]  = { .type = NLA_NUL_STRING },
+     [IFLA_IFINDEX]          = { .type = NLA_S32 },

you need to make sure this new attribute can not be used in setlink requests or getlink.

Right, I'll add it.

Quoted 8 lines
Quoted 8 lines
net/core/rtnetlink.c +0 -0
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)
      return 0;
 }
 
Quoted 5 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3057

Quoted 3 lines
net/core/rtnetlink.c +3 -0
+static int dev_ifindex_cmp(const void *a, const void *b)
+{
+     struct net_device * const *dev1 = a, * const *dev2 = b;

const struct net_device *dev1 = 1, *dev2 = b;

The array stores 'struct net_device *', and 'a' and 'b' are pointers to elements in this array (so 'struct net_device **', ignoring constness)

Quoted 56 lines
Quoted 8 lines
net/core/rtnetlink.c +7 -0
+
+     return (*dev1)->ifindex - (*dev2)->ifindex;
+}
+
+static int rtnl_ifindex_dellink(struct net *net, struct nlattr *head, int len,
+                             struct netlink_ext_ack *extack)
+{

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3074

Quoted 10 lines
net/core/rtnetlink.c +10 -0
+     int i = 0, num_devices = 0, rem;
+     struct net_device **dev_list;
+     const struct nlattr *nla;
+     LIST_HEAD(list_kill);
+     int ret;
+
+     nla_for_each_attr(nla, head, len, rem) {
+             if (nla_type(nla) == IFLA_IFINDEX)
+                     num_devices++;
+     }

The need to walk the list twice (3 really with the sort) to means the array solution is better.

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3086

Quoted 12 lines
net/core/rtnetlink.c +12 -0
+
+     dev_list = kmalloc_array(num_devices, sizeof(*dev_list), GFP_KERNEL);
+     if (!dev_list)
+             return -ENOMEM;
+
+     nla_for_each_attr(nla, head, len, rem) {
+             const struct rtnl_link_ops *ops;
+             struct net_device *dev;
+             int ifindex;
+
+             if (nla_type(nla) != IFLA_IFINDEX)
+                     continue;

and this business too. We have arrays in other places (net/ipv4/nexthop.c), so this is not the first.

Quoted 16 lines
net/core/rtnetlink.c +16 -0
+
+             ifindex = nla_get_s32(nla);
+             ret = -ENODEV;
+             dev = __dev_get_by_index(net, ifindex);
+             if (!dev) {
+                     NL_SET_ERR_MSG_ATTR(extack, nla, "Unknown ifindex");
+                     goto out_free;
+             }
+
+             ret = -EOPNOTSUPP;
+             ops = dev->rtnl_link_ops;
+             if (!ops || !ops->dellink) {
+                     NL_SET_ERR_MSG_ATTR(extack, nla, "Device cannot be deleted");
+                     goto out_free;
+             }
+
Quoted 8 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3107

Quoted 5 lines
net/core/rtnetlink.c +5 -0
+             dev_list[i++] = dev;
+     }
+
+     /* Sort devices, so we could skip duplicates */
+     sort(dev_list, num_devices, sizeof(*dev_list), dev_ifindex_cmp, NULL);

how did this sort change the results? 10k compares and re-order has to add some overhead.

No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.

Also just pointing out that the sort will be needed even if we pass an array (IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a new hlist_node to 'struct net_device' just for this)

Quoted 11 lines

net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3112

Quoted 6 lines
net/core/rtnetlink.c +5 -0
+
+     for (i = 0; i < num_devices; i++) {
+             struct net_device *dev = dev_list[i];
+
+             if (i != 0 && dev_list[i - 1]->ifindex == dev->ifindex)
                if (i && ...)

I liked the array variant better. Jakub?

#9 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

CAUTION: External E-Mail - Use caution with links and attachments

Quoted 2 lines · On Mon, 6 Dec 2021 21:12:33 -0700 David Ahern wrote:

The need to walk the list twice (3 really with the sort) to means the array solution is better.

Technically the linear walk will be dwarfed by the sort for non-trivial inputs (have I been conducting too many coding interviews?)

Quoted 7 lines

Especially with retpolines :(

Quoted 1 line

I liked the array variant better. Jakub?

I always default to attr-per-member unless there is a strong constraint on message size. Trying to extend old school netlink (TC etc) left me scarred. But we can do array, no big deal.

Thanks, I'll send a new patch with array API and all previous comments

#11 lore raw
In reply to Lahav Schlesinger
Quoted 11 lines · Lahav Schlesinger

No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.

Also just pointing out that the sort will be needed even if we pass an array (IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a new hlist_node to 'struct net_device' just for this)

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

maintainer
#12 lore raw
Quoted 17 lines · On Wed, 8 Dec 2021 16:43:28 -0700 David Ahern wrote:
Quoted 11 lines · Lahav Schlesinger

No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.

Also just pointing out that the sort will be needed even if we pass an array (IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a new hlist_node to 'struct net_device' just for this)

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

Quoted 5 lines

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.

#13 lore raw
Quoted 20 lines · Jakub Kicinski
Quoted 19 lines · On Wed, 8 Dec 2021 16:43:28 -0700 David Ahern wrote:
Quoted 11 lines · Lahav Schlesinger

No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.

Also just pointing out that the sort will be needed even if we pass an array (IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a new hlist_node to 'struct net_device' just for this)

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849

Quoted 8 lines
Quoted 3 lines

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.

isn't that used when the delink function calls unregister_netdevice_queue?

maintainer
#14 lore raw
Quoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

Quoted 5 lines
Quoted 3 lines

If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.

isn't that used when the delink function calls unregister_netdevice_queue?

Sure but all the validation is before we start calling delink, so doesn't matter?

list to_kill, queued;

for_each_attr(nest) {
	...

	dev = get_by_index(nla_get_s32(..));
	if (!dev)
		goto cleanup;
	if (!list_empty(&dev->unreg_list))
		goto cleanup;
	...

	list_add(&to_kill, &dev->unreg_list);
}
for_each_entry_safe(to_kill) {
	list_del_init(&dev->unreg_list);
	->dellink(dev, queued);
}

unreg_many(queued);

return

cleanup:

	for_each_entry_safe(to_kill) {
		list_del_init(&dev->unreg_list);
	}

No?

#15 lore raw
Quoted 36 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

Quoted 5 lines
Quoted 3 lines

If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.

isn't that used when the delink function calls unregister_netdevice_queue?

Sure but all the validation is before we start calling delink, so doesn't matter?

list to_kill, queued;

for_each_attr(nest) {
	...

	dev = get_by_index(nla_get_s32(..));
	if (!dev)
		goto cleanup;
	if (!list_empty(&dev->unreg_list))
		goto cleanup;
	...

	list_add(&to_kill, &dev->unreg_list);
}
Quoted 2 lines
for_each_entry_safe(to_kill) {
	list_del_init(&dev->unreg_list);

sure my mental model was walking the list and not doing that part. :-)

Quoted 14 lines
	->dellink(dev, queued);
}

unreg_many(queued);

return

cleanup:

	for_each_entry_safe(to_kill) {
		list_del_init(&dev->unreg_list);
	}

No?

#16 lore raw
In reply to Jakub Kicinski
Quoted 46 lines · Jakub Kicinski

CAUTION: External E-Mail - Use caution with links and attachments

Quoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines

I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_netdev_net-2Dnext.git_tree_net_ipv4_nexthop.c-23n1849&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=-WbGKi6-8WU3IfnrkySymtcZWJMn_aVdolIPebPZGu4&m=td9sv1_NnALM3QjLFg8h9u1P7DUJJL2YhzqyneZ95Gc&s=nEGgvxCwd3MrSXWhv-5Fo7mUlBkuKbHRBAO4VO9mG2o&e=

Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.

Quoted 5 lines
Quoted 3 lines

If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.

isn't that used when the delink function calls unregister_netdevice_queue?

Sure but all the validation is before we start calling delink, so doesn't matter?

list to_kill, queued;

for_each_attr(nest) {
        ...

        dev = get_by_index(nla_get_s32(..));
        if (!dev)
                goto cleanup;
        if (!list_empty(&dev->unreg_list))
                goto cleanup;
        ...

        list_add(&to_kill, &dev->unreg_list);
}
for_each_entry_safe(to_kill) {
        list_del_init(&dev->unreg_list);
        ->dellink(dev, queued);
}

unreg_many(queued);

Quoted 8 lines

return

cleanup:

        for_each_entry_safe(to_kill) {
                list_del_init(&dev->unreg_list);
        }

No?

Hi Jakub, I just sent a V6 for this patch. Just wanted to note that I tried this approach of using unreg_list, but it caused issues with e.g. veth deletion - the dellink() of a veth peer automatically adds the other peer to the queued list. So if the list of ifindices contains both peers then when 'to_kill' is iterated, both the current device and the next pointer will have their 'unreg_list' moved to the other list, which later raised a page fault when 'to_kill' was further iterated upon.

Therefore instead I added a big flag in a "hole" inside struct net_device, as David suggested.

maintainer
#17 lore raw
Quoted 11 lines · On Tue, 4 Jan 2022 10:21:21 +0200 Lahav Schlesinger wrote:

Hi Jakub, I just sent a V6 for this patch. Just wanted to note that I tried this approach of using unreg_list, but it caused issues with e.g. veth deletion - the dellink() of a veth peer automatically adds the other peer to the queued list. So if the list of ifindices contains both peers then when 'to_kill' is iterated, both the current device and the next pointer will have their 'unreg_list' moved to the other list, which later raised a page fault when 'to_kill' was further iterated upon.

Therefore instead I added a big flag in a "hole" inside struct net_device, as David suggested.

Ack. Thanks for trying it out.