rtnetlink: Support fine-grained netdevice bulk deletion
Subsystems (2)
PATCH v5 1/1 rtnetlink: Support fine-grained netdevice bulk deletion
Under large scale, some routers are required to support tens of thousands of devices at once, both physical and virtual (e.g. loopbacks, tunnels, vrfs, etc).
At times such routers are required to delete massive amounts of devices at once, such as when a factory reset is performed on the router (causing a deletion of all devices), or when a configuration is restored after an upgrade, or as a request from an operator.
Currently there are 2 means of deleting devices using Netlink:
1. Deleting a single device (either by ifindex using ifinfomsg::ifi_index, or by name using IFLA_IFNAME)
2. Delete all device that belong to a group (using IFLA_GROUP)
Deletion of devices one-by-one has poor performance on large scale of devices compared to "group deletion":
After all device are handled, netdev_run_todo() is called which calls rcu_barrier() to finish any outstanding RCU callbacks that were registered during the deletion of the device, then wait until the refcount of all the devices is 0, then perform final cleanups.
However, calling rcu_barrier() is a very costly operation, each call taking in the order of 10s of milliseconds.
When deleting a large number of device one-by-one, rcu_barrier() will be called for each device being deleted.
As an example, following benchmark deletes 10K loopback devices, all of which are UP and with only IPv6 LLA being configured:
1. Deleting one-by-one using 1 thread : 243 seconds
2. Deleting one-by-one using 10 thread: 70 seconds
3. Deleting one-by-one using 50 thread: 54 seconds
4. Deleting all using "group deletion": 30 seconds
Note that even though the deletion logic takes place under the rtnl lock, since the call to rcu_barrier() is outside the lock we gain some improvements.
But, while "group deletion" is the fastest, it is not suited for deleting large number of arbitrary devices which are unknown a head of time. Furthermore, moving large number of devices to a group is also a costly operation.
This patch adds support for passing an arbitrary list of ifindex of devices to delete with a new IFLA_IFINDEX attribute. A single message may contain multiple instances of this attribute).
This gives a more fine-grained control over which devices to delete, while still resulting in rcu_barrier() being called only once. Indeed, the timings of using this new API to delete 10K devices is the same as using the existing "group" deletion.
---
v4 -> v5
- Call ->dellink() only once for duplicated devices.
v3 -> v4
- Change single IFLA_INDEX_LIST into multiple IFLA_IFINDEX
- Fail if passing both IFLA_GROUP and at least one IFLA_IFNEX
v2 -> v3
- Rename 'ifindex_list' to 'ifindices', and pass it as int*
- Clamp 'ops' variable in second loop.
v1 -> v2
- Unset 'len' of IFLA_IFINDEX_LIST in policy.
- Use __dev_get_by_index() instead of n^2 loop.
- Return -ENODEV if any ifindex is not present.
- Saved devices in an array.
- Fix formatting.
include/uapi/linux/if_link.h | 1 + net/core/rtnetlink.c | 82 ++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+)
include/uapi/linux/if_link.h +1 -0
@@ -348,6 +348,7 @@ enum {IFLA_PARENT_DEV_NAME ,IFLA_PARENT_DEV_BUS_NAME , +IFLA_IFINDEX , __IFLA_MAX };
net/core/rtnetlink.c +82 -0
112 lines not on this page · raw message
Signature · 1 line
2.25.1
On 05/12/2021 11:36, Lahav Schlesinger wrote:
[snip]
Quoted 13 lines
But, while "group deletion" is the fastest, it is not suited for deleting large number of arbitrary devices which are unknown a head of time. Furthermore, moving large number of devices to a group is also a costly operation.
This patch adds support for passing an arbitrary list of ifindex of devices to delete with a new
IFLA_IFINDEXattribute. A single message may contain multiple instances of this attribute).
This gives a more fine-grained control over which devices to delete, while still resulting inrcu_barrier()being called only once. Indeed, the timings of using this new API to delete 10K devices is the same as using the existing "group" deletion.
Quoted 2 lines
Signed-off-by Lahav Schlesinger <[email protected]>---
[snip]
Note that Ido raised a good point[1] about group operations and their efficiency as a reply to v4 of the patch.
+CC Ido
Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.
[snip]
include/uapi/linux/if_link.h enum { · line 351
Quoted 9 lines
include/uapi/linux/if_link.h +1
-0@@ -348,6 +348,7 @@ enum {IFLA_PARENT_DEV_NAME ,IFLA_PARENT_DEV_BUS_NAME , + IFLA_IFINDEX,
nit: maybe the previous blank line sit better after this new attribute (and before __IFLA_MAX)?
Quoted 16 lines
include/uapi/linux/if_link.h +0
-0__IFLA_MAX };net/core/rtnetlink.c +1
-0@@ -37,6 +37,7 @@#include <linux/pci.h> #include <linux/etherdevice.h> #include <linux/bpf.h> +#include <linux/sort.h> #include <linux/uaccess.h>
net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884
Quoted 4 lines
net/core/rtnetlink.c +1
-0@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED }, [IFLA_NEW_IFINDEX] = NLA_POLICY_MIN(NLA_S32, 1), [IFLA_PARENT_DEV_NAME] = { .type = NLA_NUL_STRING }, + [IFLA_IFINDEX] = { .type = NLA_S32 },
Same policy than IFLA_NEW_IFINDEX to refuse negative ifindex.
Quoted 35 lines
net/core/rtnetlink.c +28
-0};static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX +1 ] = {@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; } +static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b; + +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{ +int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + } + + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev;
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092
Quoted 10 lines
net/core/rtnetlink.c +10
-0+int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ; + + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" );
It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099
Quoted 7 lines
net/core/rtnetlink.c +7
-0+ goto out_free; + } + + ret = -EOPNOTSUPP ; + ops = dev->rtnl_link_ops; +if (!ops || !ops->dellink) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Device cannot be deleted" );
Same here.
Thank you,
Nicolas
Quoted 36 lines · Nicolas Dichtel
Quoted 36 lines
net/core/rtnetlink.c +32
-0@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; } +static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b; + +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{ +int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + } + + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev; +int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ;
Quoted 8 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092
Quoted 6 lines
net/core/rtnetlink.c +6
-0+ + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" );It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".
extack infra does not allow dynamic messages. It does point to the location of the bad index, so userspace could figure it out.
Quoted 14 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099
Quoted 8 lines
net/core/rtnetlink.c +7
-0+ goto out_free; + } + + ret = -EOPNOTSUPP ; + ops = dev->rtnl_link_ops; +if (!ops || !ops->dellink) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Device cannot be deleted" );Same here.
Thank you,
Nicolas
Le 06/12/2021 à 16:20, David Ahern a écrit :
Quoted 46 lines · Nicolas Dichtel
Quoted 36 lines
Quoted 36 lines
net/core/rtnetlink.c +32
-0@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; } +static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b; + +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{ +int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + } + + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev; +int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ;Quoted 8 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092
Quoted 6 lines
net/core/rtnetlink.c +6
-0+ + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" );It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".
Quoted 2 lines
extack infra does not allow dynamic messages. It does point to the location of the bad index, so userspace could figure it out.
Oh yes, my bad.
Thank you,
Nicolas
Quoted 17 lines · Nicolas Dichtel
CAUTION: External E-Mail - Use caution with links and attachments
Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.[snip]
Quoted 8 lines
include/uapi/linux/if_link.h +0
-0@@ -348,6 +348,7 @@ enum {IFLA_PARENT_DEV_NAME , IFLA_PARENT_DEV_BUS_NAME,
Quoted 3 lines
include/uapi/linux/if_link.h enum { · line 351
Quoted 1 line
include/uapi/linux/if_link.h +1
-0+ IFLA_IFINDEX,nit: maybe the previous blank line sit better after this new attribute (and before
__IFLA_MAX)?
Due to the comment above the previous 2 attributes, I think that by removing this empty line it can be accidentally thought as if the new attribute is part of this "block".
As for adding a new line before __IFLA_MAX, I wanted to preserve the appearance we had before the IFLA_PARENT_DEV_xxx attributes where added, where there was no empty line before __IFLA_MAX.
I don't mind either way though, whatever looks better to you.
Quoted 14 lines
Quoted 14 lines
include/uapi/linux/if_link.h +0
-0__IFLA_MAX };net/core/rtnetlink.c +1
-0@@ -37,6 +37,7 @@#include <linux/pci.h> #include <linux/etherdevice.h> #include <linux/bpf.h> +#include <linux/sort.h>
Quoted 8 lines
net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884
Quoted 7 lines
net/core/rtnetlink.c +1
-0#include <linux/uaccess.h> @@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED }, [IFLA_NEW_IFINDEX] = NLA_POLICY_MIN(NLA_S32, 1), [IFLA_PARENT_DEV_NAME] = { .type = NLA_NUL_STRING }, + [IFLA_IFINDEX] = { .type = NLA_S32 },Same policy than
IFLA_NEW_IFINDEXto refuse negative ifindex.
Right, thanks
Quoted 61 lines
Quoted 36 lines
net/core/rtnetlink.c +28
-0};static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX +1 ] = {@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; } +static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b; + +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{ +int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + } + + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev;net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3092
Quoted 10 lines
net/core/rtnetlink.c +10
-0+int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ; + + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" );It would be nice to have the ifindex in the error message. This message does not give more information than "ENODEV".
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3099
Quoted 7 lines
net/core/rtnetlink.c +7
-0+ goto out_free; + } + + ret = -EOPNOTSUPP ; + ops = dev->rtnl_link_ops; +if (!ops || !ops->dellink) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Device cannot be deleted" );Same here.
Thank you,
Nicolas
Le 07/12/2021 à 13:48, Lahav Schlesinger a écrit :
Quoted 22 lines · Nicolas Dichtel
Quoted 17 lines
CAUTION: External E-Mail - Use caution with links and attachments
Le 05/12/2021 à 10:36, Lahav Schlesinger a écrit :
Some comments below, but please, keep the people who replied to previous versions of this patch in cc.[snip]
Quoted 8 lines
include/uapi/linux/if_link.h +0
-0@@ -348,6 +348,7 @@ enum {IFLA_PARENT_DEV_NAME , IFLA_PARENT_DEV_BUS_NAME,Quoted 3 lines
include/uapi/linux/if_link.h enum { · line 351
Quoted 1 line
include/uapi/linux/if_link.h +1
-0+ IFLA_IFINDEX,nit: maybe the previous blank line sit better after this new attribute (and before
__IFLA_MAX)?
Quoted 6 lines
Due to the comment above the previous 2 attributes, I think that by removing this empty line it can be accidentally thought as if the new attribute is part of this "block".
As for adding a new line before__IFLA_MAX, I wanted to preserve the appearance we had before theIFLA_PARENT_DEV_xxxattributes where added, where there was no empty line before__IFLA_MAX.
Good point.
Quoted 2 lines
I don't mind either way though, whatever looks better to you.
Ok, forget my comment.
Regards,
Nicolas
Quoted 13 lines · Lahav Schlesinger
net/core/rtnetlink.c +1
-0@@ -37,6 +37,7 @@#include <linux/pci.h> #include <linux/etherdevice.h> #include <linux/bpf.h> +#include <linux/sort.h> #include <linux/uaccess.h>
net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884
Quoted 4 lines
net/core/rtnetlink.c +1
-0@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED }, [IFLA_NEW_IFINDEX] = NLA_POLICY_MIN(NLA_S32, 1), [IFLA_PARENT_DEV_NAME] = { .type = NLA_NUL_STRING }, + [IFLA_IFINDEX] = { .type = NLA_S32 },
you need to make sure this new attribute can not be used in setlink requests or getlink.
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3057
Quoted 10 lines
net/core/rtnetlink.c +3
-0};static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX +1 ] = {@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; } +static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b;
const struct net_device *dev1 = 1, *dev2 = b;
Quoted 7 lines
net/core/rtnetlink.c +7
-0+ +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3074
Quoted 10 lines
net/core/rtnetlink.c +10
-0+int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + }
The need to walk the list twice (3 really with the sort) to means the array solution is better.
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3086
Quoted 12 lines
net/core/rtnetlink.c +12
-0+ + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev; +int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ;
and this business too. We have arrays in other places (net/ipv4/nexthop.c), so this is not the first.
Quoted 11 lines
net/core/rtnetlink.c +11
-0+ + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" ); + goto out_free; + } + + ret = -EOPNOTSUPP ; + ops = dev->rtnl_link_ops;
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3107
Quoted 10 lines
net/core/rtnetlink.c +10
-0+if (!ops || !ops->dellink) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Device cannot be deleted" ); + goto out_free; + } + + dev_list[i++] = dev; + } + +/* Sort devices, so we could skip duplicates */ +sort (dev_list, num_devices,sizeof (*dev_list), dev_ifindex_cmp,NULL );
how did this sort change the results? 10k compares and re-order has to add some overhead.
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3112
Quoted 5 lines
net/core/rtnetlink.c +5
-0+ +for (i =0 ; i < num_devices; i++) { +struct net_device *dev = dev_list[i]; + +if (i !=0 && dev_list[i -1 ]->ifindex == dev->ifindex)
if (i && ...)
I liked the array variant better. Jakub?
Jakub Kicinski
maintainerQuoted 2 lines · On Mon, 6 Dec 2021 21:12:33 -0700 David Ahern wrote:
The need to walk the list twice (3 really with the sort) to means the array solution is better.
Technically the linear walk will be dwarfed by the sort for non-trivial inputs (have I been conducting too many coding interviews?)
Especially with retpolines :(
Quoted 1 line
I liked the array variant better. Jakub?
I always default to attr-per-member unless there is a strong constraint on message size. Trying to extend old school netlink (TC etc) left me scarred. But we can do array, no big deal.
Quoted 10 lines · Jakub Kicinski
CAUTION: External E-Mail - Use caution with links and attachments
Quoted 2 lines · On Mon, 6 Dec 2021 21:12:33 -0700 David Ahern wrote:
The need to walk the list twice (3 really with the sort) to means the array solution is better.
Technically the linear walk will be dwarfed by the sort for non-trivial inputs (have I been conducting too many coding interviews?)
Quoted 7 lines
Especially with retpolines :(
Quoted 1 line
I liked the array variant better. Jakub?
I always default to attr-per-member unless there is a strong constraint on message size. Trying to extend old school netlink (TC etc) left me scarred. But we can do array, no big deal.
Thanks, I'll send a new patch with array API and all previous comments
Quoted 16 lines · David Ahern
CAUTION: External E-Mail - Use caution with links and attachments
Quoted 12 lines · Lahav Schlesinger
net/core/rtnetlink.c +1
-0@@ -37,6 +37,7 @@#include <linux/pci.h> #include <linux/etherdevice.h> #include <linux/bpf.h> +#include <linux/sort.h> #include <linux/uaccess.h>
Quoted 8 lines
net/core/rtnetlink.c static const struct nla_policy ifla_policy[IFLA_MAX+1] = { · line 1884
Quoted 5 lines
net/core/rtnetlink.c +1
-0@@ -1880,6 +1881,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED }, [IFLA_NEW_IFINDEX] = NLA_POLICY_MIN(NLA_S32, 1), [IFLA_PARENT_DEV_NAME] = { .type = NLA_NUL_STRING }, + [IFLA_IFINDEX] = { .type = NLA_S32 },you need to make sure this new attribute can not be used in setlink requests or getlink.
Right, I'll add it.
Quoted 8 lines
Quoted 8 lines
net/core/rtnetlink.c +0
-0};static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX +1 ] = {@@ -3050,6 +3052,78 @@ static int rtnl_group_dellink(const struct net *net, int group)return 0 ; }
Quoted 5 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3057
Quoted 3 lines
net/core/rtnetlink.c +3
-0+static int dev_ifindex_cmp (const void *a,const void *b) +{ +struct net_device *const *dev1 = a, *const *dev2 = b;const struct
net_device*dev1 = 1, *dev2 = b;
The array stores 'struct net_device *', and 'a' and 'b' are pointers to elements in this array (so 'struct net_device **', ignoring constness)
Quoted 56 lines
Quoted 8 lines
net/core/rtnetlink.c +7
-0+ +return (*dev1)->ifindex - (*dev2)->ifindex; +} + +static int rtnl_ifindex_dellink (struct net *net,struct nlattr *head,int len, +struct netlink_ext_ack *extack) +{net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3074
Quoted 10 lines
net/core/rtnetlink.c +10
-0+int i =0 , num_devices =0 , rem; +struct net_device **dev_list; +const struct nlattr *nla; +LIST_HEAD (list_kill); +int ret; + +nla_for_each_attr (nla, head, len, rem) { +if (nla_type (nla) ==IFLA_IFINDEX ) + num_devices++; + }The need to walk the list twice (3 really with the sort) to means the array solution is better.
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3086
Quoted 12 lines
net/core/rtnetlink.c +12
-0+ + dev_list =kmalloc_array (num_devices,sizeof (*dev_list),GFP_KERNEL ); +if (!dev_list) +return -ENOMEM ; + +nla_for_each_attr (nla, head, len, rem) { +const struct rtnl_link_ops *ops; +struct net_device *dev; +int ifindex; + +if (nla_type (nla) !=IFLA_IFINDEX ) +continue ;and this business too. We have arrays in other places (
net/ipv4/nexthop.c), so this is not the first.Quoted 16 lines
net/core/rtnetlink.c +16
-0+ + ifindex =nla_get_s32 (nla); + ret = -ENODEV ; + dev =__dev_get_by_index (net, ifindex); +if (!dev) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Unknown ifindex" ); + goto out_free; + } + + ret = -EOPNOTSUPP ; + ops = dev->rtnl_link_ops; +if (!ops || !ops->dellink) { +NL_SET_ERR_MSG_ATTR (extack, nla,"Device cannot be deleted" ); + goto out_free; + } +
Quoted 8 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3107
Quoted 5 lines
net/core/rtnetlink.c +5
-0+ dev_list[i++] = dev; + } + +/* Sort devices, so we could skip duplicates */ +sort (dev_list, num_devices,sizeof (*dev_list), dev_ifindex_cmp,NULL );how did this sort change the results? 10k compares and re-order has to add some overhead.
No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.
Also just pointing out that the sort will be needed even if we pass an array (IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a new hlist_node to 'struct net_device' just for this)
Quoted 11 lines
net/core/rtnetlink.c static int rtnl_group_dellink(const struct net *net, int group) · line 3112
Quoted 6 lines
net/core/rtnetlink.c +5
-0+ +for (i =0 ; i < num_devices; i++) { +struct net_device *dev = dev_list[i]; + +if (i !=0 && dev_list[i -1 ]->ifindex == dev->ifindex)if (i && ...)I liked the array variant better. Jakub?
Quoted 11 lines · Lahav Schlesinger
No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.Also just pointing out that the sort will be needed even if we pass an array (
IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a newhlist_nodeto 'structnet_device' just for this)
I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849
Find a hole in net_device struct in an area used only for control path and add 'bool grp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.
Jakub Kicinski
maintainerQuoted 17 lines · On Wed, 8 Dec 2021 16:43:28 -0700 David Ahern wrote:
Quoted 11 lines · Lahav Schlesinger
No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.Also just pointing out that the sort will be needed even if we pass an array (
IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a newhlist_nodeto 'structnet_device' just for this)I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
Quoted 5 lines
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849
Find a hole in
net_devicestruct in an area used only for control path and add 'boolgrp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.
If we want to keep state in the netdev itself we can probably piggy back on dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.
Quoted 20 lines · Jakub Kicinski
Quoted 19 lines · On Wed, 8 Dec 2021 16:43:28 -0700 David Ahern wrote:
Quoted 11 lines · Lahav Schlesinger
No visible changes from what I saw, this API is as fast as group deletion. Maybe a few tens of milliseconds slower, but it's lost in the noise.
I'll run more thorough benchmarks to get to a more conclusive conclusion.Also just pointing out that the sort will be needed even if we pass an array (
IFLA_IFINDEX_LIST) instead.
Feels like CS 101, but do you have a better approach for detecting duplicates in an array? I imagine a hash table will be slower as it will need to allocate a node object for each device (assuming we don't want to add a newhlist_nodeto 'structnet_device' just for this)I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849
Quoted 8 lines
Quoted 3 lines
Find a hole in
net_devicestruct in an area used only for control path and add 'boolgrp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.If we want to keep state in the netdev itself we can probably piggy back on
dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.
isn't that used when the delink function calls unregister_netdevice_queue?
Jakub Kicinski
maintainerQuoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines
I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849
Find a hole in
net_devicestruct in an area used only for control path and add 'boolgrp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.
Quoted 5 lines
Quoted 3 lines
If we want to keep state in the netdev itself we can probably piggy back on
dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.isn't that used when the delink function calls
unregister_netdevice_queue?
Sure but all the validation is before we start calling delink, so doesn't matter?
list to_kill, queued;
for_each_attr(nest) {
...
dev = get_by_index(nla_get_s32(..));
if (!dev)
goto cleanup;
if (!list_empty(&dev->unreg_list))
goto cleanup;
...
list_add(&to_kill, &dev->unreg_list);
}for_each_entry_safe(to_kill) {
list_del_init(&dev->unreg_list);
->dellink(dev, queued);
}unreg_many(queued);
return
cleanup:
for_each_entry_safe(to_kill) {
list_del_init(&dev->unreg_list);
}No?
Quoted 36 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines
I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/ipv4/nexthop.c#n1849
Find a hole in
net_devicestruct in an area used only for control path and add 'boolgrp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.Quoted 5 lines
Quoted 3 lines
If we want to keep state in the netdev itself we can probably piggy back on
dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.isn't that used when the delink function calls
unregister_netdevice_queue?Sure but all the validation is before we start calling delink, so doesn't matter?
list
to_kill, queued;for_each_attr(nest) { ... dev = get_by_index(nla_get_s32(..)); if (!dev) goto cleanup; if (!list_empty(&dev->unreg_list)) goto cleanup; ... list_add(&to_kill, &dev->unreg_list); }
Quoted 2 lines
for_each_entry_safe(to_kill) {
list_del_init(&dev->unreg_list);sure my mental model was walking the list and not doing that part. :-)
Quoted 14 lines
->dellink(dev, queued); }
unreg_many(queued);return
cleanup:
for_each_entry_safe(to_kill) { list_del_init(&dev->unreg_list); }No?
Quoted 46 lines · Jakub Kicinski
CAUTION: External E-Mail - Use caution with links and attachments
Quoted 11 lines · On Wed, 8 Dec 2021 17:18:48 -0700 David Ahern wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines
I think marking the dev's and then using a delete loop is going to be the better approach - avoid the sort and duplicate problem. I use that approach for nexthop deletes:
Find a hole in
net_devicestruct in an area used only for control path and add 'boolgrp_delete' (or a 1-bit hole). Mark the devices on pass and delete them on another.Quoted 5 lines
Quoted 3 lines
If we want to keep state in the netdev itself we can probably piggy back on
dev->unreg_list. It should be initialized to empty and not touched unless device goes thru unregister.isn't that used when the delink function calls
unregister_netdevice_queue?Sure but all the validation is before we start calling delink, so doesn't matter?
list
to_kill, queued;for_each_attr(nest) { ... dev = get_by_index(nla_get_s32(..)); if (!dev) goto cleanup; if (!list_empty(&dev->unreg_list)) goto cleanup; ... list_add(&to_kill, &dev->unreg_list); }for_each_entry_safe(to_kill) { list_del_init(&dev->unreg_list); ->dellink(dev, queued); }
unreg_many(queued);
Quoted 8 lines
return
cleanup:
for_each_entry_safe(to_kill) { list_del_init(&dev->unreg_list); }No?
Hi Jakub, I just sent a V6 for this patch. Just wanted to note that I tried this approach of using unreg_list, but it caused issues with e.g. veth deletion - the dellink() of a veth peer automatically adds the other peer to the queued list. So if the list of ifindices contains both peers then when 'to_kill' is iterated, both the current device and the next pointer will have their 'unreg_list' moved to the other list, which later raised a page fault when 'to_kill' was further iterated upon.
Therefore instead I added a big flag in a "hole" inside struct net_device, as David suggested.
Jakub Kicinski
maintainerQuoted 11 lines · On Tue, 4 Jan 2022 10:21:21 +0200 Lahav Schlesinger wrote:
Hi Jakub, I just sent a V6 for this patch. Just wanted to note that I tried this approach of using
unreg_list, but it caused issues with e.g. veth deletion - thedellink()of a veth peer automatically adds the other peer to the queued list. So if the list of ifindices contains both peers then when 'to_kill' is iterated, both the current device and the next pointer will have their 'unreg_list' moved to the other list, which later raised a page fault when 'to_kill' was further iterated upon.Therefore instead I added a big flag in a "hole" inside struct
net_device, as David suggested.
Ack. Thanks for trying it out.