devlink: remove the wait-for-references on unregister
Subsystems (4)
Jakub Kicinski
maintainercover devlink: remove the wait-for-references on unregister
Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.
Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.
This fixes the problem of the netdev code accessing devlink instances before they are registered.
RFC: https://lore.kernel.org/all/[email protected]/
- rewrite the cover letter
- rewrite the commit message for patch 1
- un-export and rename devl_is_alive
- squash the netdevsim patches
Jakub Kicinski (9):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protect devlink->dev by the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
devlink: allow registering parameters after the instance
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock
drivers/net/netdevsim/dev.c | 15 +++-- include/net/devlink.h | 2 + net/devlink/core.c | 121 ++++++++++++++++-------------------- net/devlink/devl_internal.h | 28 ++++----- net/devlink/leftover.c | 64 ++++++++++++------- net/devlink/netlink.c | 19 ++++-- 6 files changed, 137 insertions(+), 112 deletions(-)
Signature · 1 line
2.38.1
Jakub Kicinski
maintainerPATCH 1/9 devlink: bump the instance index directly when iterating
xa_find_after() is designed to handle multi-index entries correctly. If a xarray has two entries one which spans indexes 0-3 and one at index 4 xa_find_after(0) will return the entry at index 4.
Having to juggle the two callbacks, however, is unnecessary in case of the devlink xarray, as there is 1:1 relationship with indexes.
Always use xa_find() and increment the index manually.
--- net/devlink/core.c | 31 +++++++++---------------------- net/devlink/devl_internal.h | 17 ++++------------- 2 files changed, 13 insertions(+), 35 deletions(-)
net/devlink/core.c +9 -22
@@ -91,16 +91,13 @@ void devlink_put(struct devlink *devlink)call_rcu (&devlink->rcu, __devlink_put_rcu); } -struct devlink * -devlinks_xa_find_get (struct net *net,unsigned long *indexp, -void * (*xa_find_fn)(struct xarray *,unsigned long *, -unsigned long ,xa_mark_t )) +struct devlink *devlinks_xa_find_get (struct net *net,unsigned long *indexp) { -struct devlink *devlink; +struct devlink *devlink =NULL ;rcu_read_lock (); retry: - devlink =xa_find_fn (&devlinks, indexp,ULONG_MAX ,DEVLINK_REGISTERED ); + devlink =xa_find (&devlinks, indexp,ULONG_MAX ,DEVLINK_REGISTERED );if (!devlink) goto unlock;
@@ -109,31 +106,21 @@ devlinks_xa_find_get(struct net *net, unsigned long *indexp,* This prevents live-lock of devlink_unregister() wait for completion. */ if (xa_get_mark (&devlinks, *indexp,DEVLINK_UNREGISTERING )) - goto retry; + goto next; -/* For a possible retry, the xa_find_after() should be always used */ - xa_find_fn = xa_find_after;if (!devlink_try_get (devlink)) - goto retry; + goto next;if (!net_eq (devlink_net (devlink), net)) {devlink_put (devlink); - goto retry; + goto next; } unlock:rcu_read_unlock ();return devlink; -} - -struct devlink * -devlinks_xa_find_get_first (struct net *net,unsigned long *indexp) -{ -return devlinks_xa_find_get (net, indexp, xa_find); -} -struct devlink * -devlinks_xa_find_get_next (struct net *net,unsigned long *indexp) -{ -return devlinks_xa_find_get (net, indexp, xa_find_after); +next: + (*indexp)++; + goto retry; }/**
net/devlink/devl_internal.h +4 -13
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;* in loop body in order to release the reference. */ #define devlinks_xa_for_each_registered_get (net, index, devlink) \ - for (index = 0, \ - devlink = devlinks_xa_find_get_first(net, &index); \ - devlink; devlink = devlinks_xa_find_get_next(net, &index)) - -struct devlink * -devlinks_xa_find_get (struct net *net,unsigned long *indexp, -void * (*xa_find_fn)(struct xarray *,unsigned long *, -unsigned long ,xa_mark_t )); -struct devlink * -devlinks_xa_find_get_first (struct net *net,unsigned long *indexp); -struct devlink * -devlinks_xa_find_get_next (struct net *net,unsigned long *indexp); + for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++) + +struct devlink *devlinks_xa_find_get (struct net *net,unsigned long *indexp);/* Netlink */ #define DEVLINK_NL_FLAG_NEED_PORT BIT(0)
@@ -135,7 +126,7 @@ struct devlink_gen_cmd {*/ #define devlink_dump_for_each_instance_get (msg, state, devlink) \ for (; (devlink = devlinks_xa_find_get(sock_net(msg->sk), \ - &state->instance, xa_find)); \ + &state->instance)); \ state->instance++, state->idx = 0)extern const struct genl_small_ops devlink_nl_ops[56 ];
Signature · 1 line
2.38.1
Re: [PATCH net-next 1/9] devlink: bump the instance index directly when iterating
Quoted 10 lines · Fri, Jan 06, 2023 at 07:33:54AM CET, [email protected] wrote:
xa_find_after()is designed to handle multi-index entries correctly. If a xarray has two entries one which spans indexes 0-3 and one at index 4xa_find_after(0)will return the entry at index 4.Having to juggle the two callbacks, however, is unnecessary in case of the devlink xarray, as there is 1:1 relationship with indexes.
Always use
xa_find()and increment the index manually.Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 2/9 devlink: update the code in netns move to latest helpers
In reply to Jakub Kicinskidevlink_pernet_pre_exit() is the only obvious place which takes the instance lock without using the devl_ helpers. Update the code and move the error print after releasing the reference (having unlock and put together feels slightly idiomatic).
--- net/devlink/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
net/devlink/core.c +4 -3
@@ -299,15 +299,16 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)*/ devlinks_xa_for_each_registered_get (net, index, devlink) {WARN_ON (!(devlink->features &DEVLINK_F_RELOAD )); -mutex_lock (&devlink->lock); +devl_lock (devlink); err =devlink_reload (devlink, &init_net,DEVLINK_RELOAD_ACTION_DRIVER_REINIT ,DEVLINK_RELOAD_LIMIT_UNSPEC , &actions_performed,NULL ); -mutex_unlock (&devlink->lock); +devl_unlock (devlink); +devlink_put (devlink); +if (err && err != -EOPNOTSUPP )pr_warn ("Failed to reload devlink instance into init_net\n" ); -devlink_put (devlink); } }
Signature · 1 line
2.38.1
Jakub Kicinski
maintainerPATCH 3/9 devlink: protect devlink->dev by the instance lock
In reply to Jakub Kicinskidevlink->dev is assumed to be always valid as long as any outstanding reference to the devlink instance exists.
In prep for weakening of the references take the instance lock.
--- net/devlink/devl_internal.h | 3 ++- net/devlink/leftover.c | 7 +++---- net/devlink/netlink.c | 9 ++++++--- 3 files changed, 11 insertions(+), 8 deletions(-)
net/devlink/devl_internal.h +2 -1
@@ -131,7 +131,8 @@ struct devlink_gen_cmd {extern const struct genl_small_ops devlink_nl_ops[56 ]; -struct devlink *devlink_get_from_attrs (struct net *net,struct nlattr **attrs); +struct devlink * +devlink_get_from_attrs_lock (struct net *net,struct nlattr **attrs);void devlink_notify_unregister (struct devlink *devlink);void devlink_notify_register (struct devlink *devlink);
net/devlink/leftover.c +3 -4
@@ -6314,12 +6314,10 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, start_offset = state->start_offset; - devlink =devlink_get_from_attrs (sock_net (cb->skb->sk), attrs); + devlink =devlink_get_from_attrs_lock (sock_net (cb->skb->sk), attrs);if (IS_ERR (devlink))return PTR_ERR (devlink); -devl_lock (devlink); -if (!attrs[DEVLINK_ATTR_REGION_NAME ]) {NL_SET_ERR_MSG (cb->extack,"No region name provided" ); err = -EINVAL ;
@@ -7735,9 +7733,10 @@ devlink_health_reporter_get_from_cb(struct netlink_callback *cb)struct nlattr **attrs = info->attrs;struct devlink *devlink; - devlink =devlink_get_from_attrs (sock_net (cb->skb->sk), attrs); + devlink =devlink_get_from_attrs_lock (sock_net (cb->skb->sk), attrs);if (IS_ERR (devlink))return NULL ; +devl_unlock (devlink); reporter =devlink_health_reporter_get_from_attrs (devlink, attrs);devlink_put (devlink);
net/devlink/netlink.c +6 -3
@@ -82,7 +82,8 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_REGION_DIRECT] = { .type = NLA_FLAG }, }; -struct devlink *devlink_get_from_attrs (struct net *net,struct nlattr **attrs) +struct devlink * +devlink_get_from_attrs_lock (struct net *net,struct nlattr **attrs) {struct devlink *devlink;unsigned long index;
@@ -96,9 +97,11 @@ struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs) devname =nla_data (attrs[DEVLINK_ATTR_DEV_NAME ]);devlinks_xa_for_each_registered_get (net, index, devlink) { +devl_lock (devlink);if (strcmp (devlink->dev->bus->name, busname) ==0 &&strcmp (dev_name (devlink->dev), devname) ==0 )return devlink; +devl_unlock (devlink);devlink_put (devlink); }
@@ -113,10 +116,10 @@ static int devlink_nl_pre_doit(const struct genl_split_ops *ops,struct devlink *devlink;int err; - devlink =devlink_get_from_attrs (genl_info_net (info), info->attrs); + devlink =devlink_get_from_attrs_lock (genl_info_net (info), info->attrs);if (IS_ERR (devlink))return PTR_ERR (devlink); -devl_lock (devlink); + info->user_ptr[0 ] = devlink;if (ops->internal_flags &DEVLINK_NL_FLAG_NEED_PORT ) { devlink_port =devlink_port_get_from_info (devlink, info);
Signature · 1 line
2.38.1
Re: [PATCH net-next 3/9] devlink: protect devlink->dev by the instance lock
Quoted 6 lines · Fri, Jan 06, 2023 at 07:33:56AM CET, [email protected] wrote:
devlink->devis assumed to be always valid as long as any outstanding reference to the devlink instance exists.In prep for weakening of the references take the instance lock.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 4/9 devlink: always check if the devlink instance is registered
In reply to Jakub KicinskiAlways check under the instance lock whether the devlink instance is still / already registered.
This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.
Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.
--- net/devlink/core.c | 19 +++++++++++++++---- net/devlink/devl_internal.h | 8 ++++++++ net/devlink/leftover.c | 35 +++++++++++++++++++++++++++++------ net/devlink/netlink.c | 10 ++++++++-- 4 files changed, 60 insertions(+), 12 deletions(-)
net/devlink/core.c +15 -4
@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink) }EXPORT_SYMBOL_GPL (devl_unlock); +/** +* devlink_try_get() - try to obtain a reference on a devlink instance +* @devlink: instance to reference +* +* Obtain a reference on a devlink instance. A reference on a devlink instance +* only implies that it's safe to take the instance lock. It does not imply +* that the instance is registered, use devl_is_registered() after taking +* the instance lock to check registration status. +*/ struct devlink *__must_check devlink_try_get (struct devlink *devlink) {if (refcount_inc_not_zero (&devlink->refcount))
@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)devlinks_xa_for_each_registered_get (net, index, devlink) {WARN_ON (!(devlink->features &DEVLINK_F_RELOAD ));devl_lock (devlink); - err =devlink_reload (devlink, &init_net, -DEVLINK_RELOAD_ACTION_DRIVER_REINIT , -DEVLINK_RELOAD_LIMIT_UNSPEC , - &actions_performed,NULL ); + err =0 ; +if (devl_is_registered (devlink)) + err =devlink_reload (devlink, &init_net, +DEVLINK_RELOAD_ACTION_DRIVER_REINIT , +DEVLINK_RELOAD_LIMIT_UNSPEC , + &actions_performed,NULL );devl_unlock (devlink);devlink_put (devlink);
net/devlink/devl_internal.h +8 -0
@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;struct devlink *devlinks_xa_find_get (struct net *net,unsigned long *indexp); +static inline bool devl_is_registered (struct devlink *devlink) +{ +/* To prevent races the caller must hold the instance lock +* or another lock taken during unregistration. +*/ +return xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED ); +} +/* Netlink */ #define DEVLINK_NL_FLAG_NEED_PORT BIT(0)#define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1)
net/devlink/leftover.c +29 -6
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->linecards_lock); +if (!devl_is_registered (devlink)) + goto next_devlink; +list_for_each_entry (linecard, &devlink->linecard_list, list) {if (idx < state->idx) { idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, } idx++; } +next_devlink:mutex_unlock (&devlink->linecards_lock);devlink_put (devlink); }
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->reporters_lock); +if (!devl_is_registered (devlink)) { +mutex_unlock (&devlink->reporters_lock); +devlink_put (devlink); +continue ; + } +list_for_each_entry (reporter, &devlink->reporter_list, list) {if (idx < state->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,mutex_unlock (&devlink->reporters_lock);devl_lock (devlink); +if (!devl_is_registered (devlink)) + goto next_devlink; +xa_for_each (&devlink->ports, port_index, port) {mutex_lock (&port->reporters_lock);list_for_each_entry (reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, }mutex_unlock (&port->reporters_lock); } +next_devlink:devl_unlock (devlink);devlink_put (devlink); }
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,return ;devl_lock (devlink); -__devlink_compat_running_version (devlink, buf, len); +if (devl_is_registered (devlink)) +__devlink_compat_running_version (devlink, buf, len);devl_unlock (devlink); }
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)struct devlink_flash_update_params params = {};int ret; -if (!devlink->ops->flash_update) -return -EOPNOTSUPP ; +devl_lock (devlink); +if (!devl_is_registered (devlink)) { + ret = -ENODEV ; + goto out_unlock; + } + +if (!devlink->ops->flash_update) { + ret = -EOPNOTSUPP ; + goto out_unlock; + } ret =request_firmware (¶ms.fw, file_name, devlink->dev);if (ret) -return ret; + goto out_unlock; -devl_lock (devlink);devlink_flash_update_begin_notify (devlink); ret = devlink->ops->flash_update (devlink, ¶ms,NULL );devlink_flash_update_end_notify (devlink); -devl_unlock (devlink);release_firmware (params.fw); +out_unlock: +devl_unlock (devlink);return ret; }
net/devlink/netlink.c +8 -2
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)devlinks_xa_for_each_registered_get (net, index, devlink) {devl_lock (devlink); -if (strcmp (devlink->dev->bus->name, busname) ==0 && +if (devl_is_registered (devlink) && +strcmp (devlink->dev->bus->name, busname) ==0 &&strcmp (dev_name (devlink->dev), devname) ==0 )return devlink;devl_unlock (devlink);
@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,devlink_dump_for_each_instance_get (msg, state, devlink) {devl_lock (devlink); - err = cmd->dump_one (msg, devlink, cb); + +if (devl_is_registered (devlink)) + err = cmd->dump_one (msg, devlink, cb); +else + err =0 ; +devl_unlock (devlink);devlink_put (devlink);
Signature · 1 line
2.38.1
Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered
Quoted 100 lines · Fri, Jan 06, 2023 at 07:33:57AM CET, [email protected] wrote:
Always check under the instance lock whether the devlink instance is still / already registered.
This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.
Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.
Signed-off-by Jakub Kicinski <[email protected]>---
net/devlink/core.c | 19 +++++++++++++++---- net/devlink/devl_internal.h | 8 ++++++++ net/devlink/leftover.c | 35 +++++++++++++++++++++++++++++------ net/devlink/netlink.c | 10 ++++++++-- 4 files changed, 60 insertions(+), 12 deletions(-)net/devlink/core.c +15
-4@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink) }EXPORT_SYMBOL_GPL (devl_unlock); +/** +* devlink_try_get() - try to obtain a reference on a devlink instance +* @devlink: instance to reference +* +* Obtain a reference on a devlink instance. A reference on a devlink instance +* only implies that it's safe to take the instance lock. It does not imply +* that the instance is registered, use devl_is_registered() after taking +* the instance lock to check registration status. +*/ struct devlink *__must_check devlink_try_get (struct devlink *devlink) {if (refcount_inc_not_zero (&devlink->refcount))@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)devlinks_xa_for_each_registered_get (net, index, devlink) {WARN_ON (!(devlink->features &DEVLINK_F_RELOAD ));devl_lock (devlink); - err =devlink_reload (devlink, &init_net, -DEVLINK_RELOAD_ACTION_DRIVER_REINIT , -DEVLINK_RELOAD_LIMIT_UNSPEC , - &actions_performed,NULL ); + err =0 ; +if (devl_is_registered (devlink)) + err =devlink_reload (devlink, &init_net, +DEVLINK_RELOAD_ACTION_DRIVER_REINIT , +DEVLINK_RELOAD_LIMIT_UNSPEC , + &actions_performed,NULL );devl_unlock (devlink);devlink_put (devlink);net/devlink/devl_internal.h +8
-0@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;struct devlink *devlinks_xa_find_get (struct net *net,unsigned long *indexp); +static inline bool devl_is_registered (struct devlink *devlink) +{ +/* To prevent races the caller must hold the instance lock +* or another lock taken during unregistration. +*/ +return xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED ); +} +/* Netlink */ #define DEVLINK_NL_FLAG_NEED_PORT BIT(0)#define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1)net/devlink/leftover.c +4
-0@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->linecards_lock); +if (!devl_is_registered (devlink)) + goto next_devlink; +list_for_each_entry (linecard, &devlink->linecard_list, list) {if (idx < state->idx) { idx++;@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, } idx++; } +next_devlink:mutex_unlock (&devlink->linecards_lock);devlink_put (devlink); }
net/devlink/leftover.c devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, · line 7816
Quoted 4 lines
net/devlink/leftover.c +1
-0@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->reporters_lock); +if (!devl_is_registered (devlink)) {
Good. I have patchset to remove this and linecard lock prepared. That makes things smoother.
Quoted 101 lines
net/devlink/leftover.c +24
-6+mutex_unlock (&devlink->reporters_lock); +devlink_put (devlink); +continue ; + } +list_for_each_entry (reporter, &devlink->reporter_list, list) {if (idx < state->idx) {@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,mutex_unlock (&devlink->reporters_lock);devl_lock (devlink); +if (!devl_is_registered (devlink)) + goto next_devlink; +xa_for_each (&devlink->ports, port_index, port) {mutex_lock (&port->reporters_lock);list_for_each_entry (reporter, &port->reporter_list, list) {@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, }mutex_unlock (&port->reporters_lock); } +next_devlink:devl_unlock (devlink);devlink_put (devlink); }@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,return ;devl_lock (devlink); -__devlink_compat_running_version (devlink, buf, len); +if (devl_is_registered (devlink)) +__devlink_compat_running_version (devlink, buf, len);devl_unlock (devlink); }@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)struct devlink_flash_update_params params = {};int ret; -if (!devlink->ops->flash_update) -return -EOPNOTSUPP ; +devl_lock (devlink); +if (!devl_is_registered (devlink)) { + ret = -ENODEV ; + goto out_unlock; + } + +if (!devlink->ops->flash_update) { + ret = -EOPNOTSUPP ; + goto out_unlock; + } ret =request_firmware (¶ms.fw, file_name, devlink->dev);if (ret) -return ret; + goto out_unlock; -devl_lock (devlink);devlink_flash_update_begin_notify (devlink); ret = devlink->ops->flash_update (devlink, ¶ms,NULL );devlink_flash_update_end_notify (devlink); -devl_unlock (devlink);release_firmware (params.fw); +out_unlock: +devl_unlock (devlink);return ret; }net/devlink/netlink.c +8
-2@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)devlinks_xa_for_each_registered_get (net, index, devlink) {devl_lock (devlink); -if (strcmp (devlink->dev->bus->name, busname) ==0 && +if (devl_is_registered (devlink) && +strcmp (devlink->dev->bus->name, busname) ==0 &&strcmp (dev_name (devlink->dev), devname) ==0 )return devlink;devl_unlock (devlink);@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,devlink_dump_for_each_instance_get (msg, state, devlink) {devl_lock (devlink); - err = cmd->dump_one (msg, devlink, cb); + +if (devl_is_registered (devlink)) + err = cmd->dump_one (msg, devlink, cb); +else + err =0 ; +devl_unlock (devlink);devlink_put (devlink);Signature · 1 line
2.38.1
Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered
In reply to Jakub KicinskiQuoted 63 lines · Jakub Kicinski
Always check under the instance lock whether the devlink instance is still / already registered.
This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.
Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.
Signed-off-by Jakub Kicinski <[email protected]>--- net/devlink/core.c | 19 +++++++++++++++---- net/devlink/devl_internal.h | 8 ++++++++ net/devlink/leftover.c | 35 +++++++++++++++++++++++++++++------ net/devlink/netlink.c | 10 ++++++++-- 4 files changed, 60 insertions(+), 12 deletions(-)net/devlink/core.c +15
-4@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink) }EXPORT_SYMBOL_GPL (devl_unlock); +/** +* devlink_try_get() - try to obtain a reference on a devlink instance +* @devlink: instance to reference +* +* Obtain a reference on a devlink instance. A reference on a devlink instance +* only implies that it's safe to take the instance lock. It does not imply +* that the instance is registered, use devl_is_registered() after taking +* the instance lock to check registration status. +*/ struct devlink *__must_check devlink_try_get (struct devlink *devlink) {if (refcount_inc_not_zero (&devlink->refcount))@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)devlinks_xa_for_each_registered_get (net, index, devlink) {WARN_ON (!(devlink->features &DEVLINK_F_RELOAD ));devl_lock (devlink); - err =devlink_reload (devlink, &init_net, -DEVLINK_RELOAD_ACTION_DRIVER_REINIT , -DEVLINK_RELOAD_LIMIT_UNSPEC , - &actions_performed,NULL ); + err =0 ; +if (devl_is_registered (devlink)) + err =devlink_reload (devlink, &init_net, +DEVLINK_RELOAD_ACTION_DRIVER_REINIT , +DEVLINK_RELOAD_LIMIT_UNSPEC , + &actions_performed,NULL );devl_unlock (devlink);devlink_put (devlink);net/devlink/devl_internal.h +0
-0
net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93
Quoted 8 lines
net/devlink/devl_internal.h +5
-0@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;struct devlink *devlinks_xa_find_get (struct net *net,unsigned long *indexp); +static inline bool devl_is_registered (struct devlink *devlink) +{ +/* To prevent races the caller must hold the instance lock +* or another lock taken during unregistration. +*/
Why not just lockdep_assert here on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like the linecards_lock as opposed to the instance lock?
Thanks,
Jake
Quoted 131 lines
net/devlink/devl_internal.h +3
-0+return xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED ); +} +/* Netlink */ #define DEVLINK_NL_FLAG_NEED_PORT BIT(0)#define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1)net/devlink/leftover.c +29
-6@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->linecards_lock); +if (!devl_is_registered (devlink)) + goto next_devlink; +list_for_each_entry (linecard, &devlink->linecard_list, list) {if (idx < state->idx) { idx++;@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, } idx++; } +next_devlink:mutex_unlock (&devlink->linecards_lock);devlink_put (devlink); }@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,int idx =0 ;mutex_lock (&devlink->reporters_lock); +if (!devl_is_registered (devlink)) { +mutex_unlock (&devlink->reporters_lock); +devlink_put (devlink); +continue ; + } +list_for_each_entry (reporter, &devlink->reporter_list, list) {if (idx < state->idx) {@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,mutex_unlock (&devlink->reporters_lock);devl_lock (devlink); +if (!devl_is_registered (devlink)) + goto next_devlink; +xa_for_each (&devlink->ports, port_index, port) {mutex_lock (&port->reporters_lock);list_for_each_entry (reporter, &port->reporter_list, list) {@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, }mutex_unlock (&port->reporters_lock); } +next_devlink:devl_unlock (devlink);devlink_put (devlink); }@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,return ;devl_lock (devlink); -__devlink_compat_running_version (devlink, buf, len); +if (devl_is_registered (devlink)) +__devlink_compat_running_version (devlink, buf, len);devl_unlock (devlink); }@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)struct devlink_flash_update_params params = {};int ret; -if (!devlink->ops->flash_update) -return -EOPNOTSUPP ; +devl_lock (devlink); +if (!devl_is_registered (devlink)) { + ret = -ENODEV ; + goto out_unlock; + } + +if (!devlink->ops->flash_update) { + ret = -EOPNOTSUPP ; + goto out_unlock; + } ret =request_firmware (¶ms.fw, file_name, devlink->dev);if (ret) -return ret; + goto out_unlock; -devl_lock (devlink);devlink_flash_update_begin_notify (devlink); ret = devlink->ops->flash_update (devlink, ¶ms,NULL );devlink_flash_update_end_notify (devlink); -devl_unlock (devlink);release_firmware (params.fw); +out_unlock: +devl_unlock (devlink);return ret; }net/devlink/netlink.c +8
-2@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)devlinks_xa_for_each_registered_get (net, index, devlink) {devl_lock (devlink); -if (strcmp (devlink->dev->bus->name, busname) ==0 && +if (devl_is_registered (devlink) && +strcmp (devlink->dev->bus->name, busname) ==0 &&strcmp (dev_name (devlink->dev), devname) ==0 )return devlink;devl_unlock (devlink);@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,devlink_dump_for_each_instance_get (msg, state, devlink) {devl_lock (devlink); - err = cmd->dump_one (msg, devlink, cb); + +if (devl_is_registered (devlink)) + err = cmd->dump_one (msg, devlink, cb); +else + err =0 ; +devl_unlock (devlink);devlink_put (devlink);
Jakub Kicinski
maintainerRe: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered
Quoted 10 lines · On Fri, 6 Jan 2023 09:03:18 -0800 Jacob Keller wrote:
net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93
Quoted 5 lines
net/devlink/devl_internal.h +5
-0+static inline bool devl_is_registered (struct devlink *devlink) +{ +/* To prevent races the caller must hold the instance lock +* or another lock taken during unregistration. +*/ Why not just
lockdep_asserthere on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like thelinecards_lockas opposed to the instance lock?
Yup, as discussed on the RFC - removing the regions lock specifically is quite tricky.
Jiri Pirko
maintainerRe: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered
Quoted 12 lines · Fri, Jan 06, 2023 at 10:19:34PM CET, [email protected] wrote:
Quoted 10 lines · On Fri, 6 Jan 2023 09:03:18 -0800 Jacob Keller wrote:
net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93
Quoted 5 lines
net/devlink/devl_internal.h +5
-0+static inline bool devl_is_registered (struct devlink *devlink) +{ +/* To prevent races the caller must hold the instance lock +* or another lock taken during unregistration. +*/ Why not just
lockdep_asserthere on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like thelinecards_lockas opposed to the instance lock?
Quoted 2 lines
Yup, as discussed on the RFC - removing the regions lock specifically is quite tricky.
I will submit in a jiff. Will add the assert here as a part of the patchset.
Jakub Kicinski
maintainerPATCH 5/9 devlink: remove the registration guarantee of references
In reply to Jakub KicinskiThe objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because devlink_unregister() waits for all references to be released, meaning that devlink_unregister() can't itself be called under the instance lock.
To avoid this issue devlink_register() was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error until devlink_register() is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).
Weaken the guarantees of the devlink references.
Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.
This implies that releasing of the "main" reference of the devlink instance moves from devlink_unregister() to devlink_free().
--- include/net/devlink.h | 2 ++ net/devlink/core.c | 64 ++++++++++++++++--------------------- net/devlink/devl_internal.h | 2 -- 3 files changed, 30 insertions(+), 38 deletions(-)
include/net/devlink.h +2 -0
@@ -1647,6 +1647,8 @@ static inline struct devlink *devlink_alloc(const struct devlink_ops *ops,return devlink_alloc_ns (ops, priv_size, &init_net, dev); }void devlink_set_features (struct devlink *devlink,u64 features); +int devl_register (struct devlink *devlink); +void devl_unregister (struct devlink *devlink);void devlink_register (struct devlink *devlink);void devlink_unregister (struct devlink *devlink);void devlink_free (struct devlink *devlink);
net/devlink/core.c +28 -36
113 lines not on this page · raw message
net/devlink/devl_internal.h +0 -2
@@ -12,7 +12,6 @@#include <net/net_namespace.h> #define DEVLINK_REGISTERED XA_MARK_1 -#define DEVLINK_UNREGISTERING XA_MARK_2#define DEVLINK_RELOAD_STATS_ARRAY_SIZE \ (__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
@@ -52,7 +51,6 @@ struct devlink {struct lock_class_key lock_key; u8 reload_failed:1;refcount_t refcount; -struct completion comp;struct rcu_head rcu;struct notifier_block netdevice_nb;char priv[]__aligned(NETDEV_ALIGN) ;
Signature · 1 line
2.38.1
Re: [PATCH net-next 5/9] devlink: remove the registration guarantee of references
Quoted 25 lines · Fri, Jan 06, 2023 at 07:33:58AM CET, [email protected] wrote:
The objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because
devlink_unregister()waits for all references to be released, meaning thatdevlink_unregister()can't itself be called under the instance lock.To avoid this issue
devlink_register()was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error untildevlink_register()is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).Weaken the guarantees of the devlink references.
Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.
Quoted 5 lines
This implies that releasing of the "main" reference of the devlink instance moves from
devlink_unregister()todevlink_free().Reviewed-by Jacob Keller <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 6/9 devlink: don't require setting features before registration
In reply to Jakub KicinskiRequiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.
--- net/devlink/core.c | 2 -- 1 file changed, 2 deletions(-)
net/devlink/core.c +0 -2
@@ -125,8 +125,6 @@ struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)*/ void devlink_set_features (struct devlink *devlink,u64 features) { -ASSERT_DEVLINK_NOT_REGISTERED (devlink); -WARN_ON (features &DEVLINK_F_RELOAD && !devlink_reload_supported (devlink->ops)); devlink->features = features;
Signature · 1 line
2.38.1
Re: [PATCH net-next 6/9] devlink: don't require setting features before registration
Quoted 6 lines · Fri, Jan 06, 2023 at 07:33:59AM CET, [email protected] wrote:
Requiring
devlink_set_features()to be run before devlink is registered is overzealous.devlink_set_features()itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.Signed-off-by Jakub Kicinski <[email protected]>
I have a patch prepared to remove this entirely. Until then, this is fine.
Jakub Kicinski
maintainerPATCH 7/9 devlink: allow registering parameters after the instance
In reply to Jakub KicinskiIt's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.
--- net/devlink/leftover.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
net/devlink/leftover.c +11 -11
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed +* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ; msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL );if (!msg)
@@ -10915,8 +10921,6 @@ int devlink_params_register(struct devlink *devlink,const struct devlink_param *param = params;int i, err; -ASSERT_DEVLINK_NOT_REGISTERED (devlink); -for (i =0 ; i < params_count; i++, param++) { err =devlink_param_register (devlink, param);if (err)
@@ -10947,8 +10951,6 @@ void devlink_params_unregister(struct devlink *devlink,const struct devlink_param *param = params;int i; -ASSERT_DEVLINK_NOT_REGISTERED (devlink); -for (i =0 ; i < params_count; i++, param++)devlink_param_unregister (devlink, param); }
@@ -10968,8 +10970,6 @@ int devlink_param_register(struct devlink *devlink, {struct devlink_param_item *param_item; -ASSERT_DEVLINK_NOT_REGISTERED (devlink); -WARN_ON (devlink_param_verify (param));WARN_ON (devlink_param_find_by_name (&devlink->param_list, param->name));
@@ -10985,6 +10985,7 @@ int devlink_param_register(struct devlink *devlink, param_item->param = param;list_add_tail (¶m_item->list, &devlink->param_list); +devlink_param_notify (devlink,0 , param_item,DEVLINK_CMD_PARAM_NEW );return 0 ; }EXPORT_SYMBOL_GPL (devlink_param_register);
@@ -10999,11 +11000,10 @@ void devlink_param_unregister(struct devlink *devlink, {struct devlink_param_item *param_item; -ASSERT_DEVLINK_NOT_REGISTERED (devlink); - param_item =devlink_param_find_by_name (&devlink->param_list, param->name);WARN_ON (!param_item); +devlink_param_notify (devlink,0 , param_item,DEVLINK_CMD_PARAM_DEL );list_del (¶m_item->list);kfree (param_item); }
@@ -11063,8 +11063,6 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id, {struct devlink_param_item *param_item; -ASSERT_DEVLINK_NOT_REGISTERED (devlink); - param_item =devlink_param_find_by_id (&devlink->param_list, param_id);if (!param_item)return -EINVAL ;
@@ -11078,6 +11076,8 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id,else param_item->driverinit_value = init_val; param_item->driverinit_value_valid = true; + +devlink_param_notify (devlink,0 , param_item,DEVLINK_CMD_PARAM_NEW );return 0 ; }EXPORT_SYMBOL_GPL (devlink_param_driverinit_value_set);
Signature · 1 line
2.38.1
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 15 lines · Fri, Jan 06, 2023 at 07:34:00AM CET, [email protected] wrote:
It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.
Signed-off-by Jakub Kicinski <[email protected]>---
net/devlink/leftover.c| 22 +++++++++++-----------1 file changed, 11 insertions(+), 11 deletions(-)net/devlink/leftover.c +0
-0@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink, WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272
Quoted 10 lines
net/devlink/leftover.c +7
-1cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd != DEVLINK_CMD_PORT_PARAM_DEL); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed +* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ;
This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.
Jakub Kicinski
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3
-1@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed
Quoted 8 lines
net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272
Quoted 4 lines
net/devlink/leftover.c +4
-0+* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ;This helper would be nice to use on other places as well. Like
devlink_trap_group_notify(),devlink_trap_notify()and others. I will take care of that in a follow-up.
Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 18 lines · Fri, Jan 06, 2023 at 10:22:51PM CET, [email protected] wrote:
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3
-1@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed Quoted 8 lines
net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272
Quoted 4 lines
net/devlink/leftover.c +4
-0+* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ;This helper would be nice to use on other places as well. Like
devlink_trap_group_notify(),devlink_trap_notify()and others. I will take care of that in a follow-up.
Quoted 3 lines
Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S
Hmm, let me explore that path. Thanks!
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
In reply to Jakub KicinskiQuoted 18 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3
-1@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed Quoted 8 lines
net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272
Quoted 4 lines
net/devlink/leftover.c +4
-0+* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ;This helper would be nice to use on other places as well. Like
devlink_trap_group_notify(),devlink_trap_notify()and others. I will take care of that in a follow-up.
Quoted 3 lines
Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S
I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 21 lines · Tue, Jan 10, 2023 at 01:21:18AM CET, [email protected] wrote:
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272
Quoted 12 lines
net/devlink/leftover.c +7
-1@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed +* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ;This helper would be nice to use on other places as well. Like
devlink_trap_group_notify(),devlink_trap_notify()and others. I will take care of that in a follow-up.
Quoted 7 lines
Quoted 3 lines
Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S
I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.
I'm confused. You want to register objects after instance register?
Jakub Kicinski
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 5 lines · On Tue, 10 Jan 2023 17:35:35 +0100 Jiri Pirko wrote:
Quoted 3 lines
I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 8 lines · Tue, Jan 10, 2023 at 09:22:22PM CET, [email protected] wrote:
Quoted 5 lines · On Tue, 10 Jan 2023 17:35:35 +0100 Jiri Pirko wrote:
Quoted 3 lines
I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Jakub Kicinski
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
Thanks
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
Quoted 7 lines
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Perhaps during reload we should emulate complete fini/init notification flow to the user?
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 8 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Perhaps during reload we should emulate complete fini/init notification flow to the user?
"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.
Thanks
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 48 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 47 lines · Jiri Pirko
Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
Quoted 7 lines
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Quoted 6 lines
Quoted 2 lines
Perhaps during reload we should emulate complete fini/init notification flow to the user?
"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.
Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.
Quoted 2 lines
Thanks
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 52 lines · Jiri Pirko
Quoted 51 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 49 lines · Jiri Pirko
Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
Quoted 7 lines
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Perhaps during reload we should emulate complete fini/init notification flow to the user?
Quoted 7 lines
Quoted 3 lines
"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.
Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.
Listen yes, however it is not clear if notification sequence should mimic fini/init flow.
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 57 lines · Sun, Jan 15, 2023 at 09:35:57AM CET, [email protected] wrote:
Quoted 56 lines · Jiri Pirko
Quoted 54 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 8 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Perhaps during reload we should emulate complete fini/init notification flow to the user?
"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.
Quoted 6 lines
Quoted 3 lines
Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.
Listen yes, however it is not clear if notification sequence should mimic fini/init flow.
Well, it makes sense to me. Why do you think it should not?
Quoted 6 lines
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 58 lines · Jiri Pirko
Quoted 57 lines · Sun, Jan 15, 2023 at 09:35:57AM CET, [email protected] wrote:
Quoted 56 lines · Jiri Pirko
Quoted 54 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line
I'm confused. You want to register objects after instance register?
+1, I think it's an anti-pattern.
Could you elaborate a bit please?
Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.
I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.
But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?
We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.
Quoted 6 lines
But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 8 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.
Perhaps during reload we should emulate complete fini/init notification flow to the user?
"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.
Listen yes, however it is not clear if notification sequence should mimic fini/init flow.
Well, it makes sense to me. Why do you think it should not?
After all this years, I still don't understand the mandate of devlink reload. It doesn't load/unload driver completely and as such not really performs probe/remove sequences. There is no requirement from the driver to do anything even close to fini/init too.
Sometimes, devlink reload behaves as fini/init, but not always.
This is why I'm not sure.
Thanks
Quoted 7 lines
Quoted 7 lines
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks
Jakub Kicinski
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
In reply to Leon RomanovskyQuoted 7 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.
I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Agree with Jakub, it is anti-pattern.
To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.
Quoted 6 lines
I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.
The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.
If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.
Thanks
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.
Right. I think its easier to simply require devlink to be registered first.
Quoted 13 lines
Quoted 6 lines
I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.
The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.
Quoted 4 lines
If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.
Thanks
Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").
Thanks,
Jake
Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 11 lines · Jacob Keller
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.
Back then, we had only one special case - netdevsim. I still think that all recent complexity that was brought to the devlink could be avoided if we would change netdevsim to behave as HW driver (remove sysfs).
Quoted 1 line
Right. I think its easier to simply require devlink to be registered first.
devlink_register() is no more than a fancy way to say to the world: "I'm ready to accept commands". Right now, when the need_lock flag is removed from all devlink commands, we can place devlink_register() at any place.
Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.
The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.
If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.
Quoted 6 lines
Quoted 1 line
Thanks
Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").
Exactly.
Quoted 3 lines
Thanks,
Jake
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
Quoted 46 lines · Fri, Jan 13, 2023 at 07:45:46AM CET, [email protected] wrote:
Quoted 11 lines · Jacob Keller
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:
As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Agree with Jakub, it is anti-pattern.
To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.
Back then, we had only one special case - netdevsim. I still think that all recent complexity that was brought to the devlink could be avoided if we would change netdevsim to behave as HW driver (remove sysfs).
Quoted 1 line
Right. I think its easier to simply require devlink to be registered first.
devlink_register()is no more than a fancy way to say to the world: "I'm ready to accept commands". Right now, when theneed_lockflag is removed from all devlink commands, we can placedevlink_register()at any place.Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.
The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.
If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Thanks
Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").
Exactly.
And reload as well.
Quoted 4 lines
Quoted 4 lines
Thanks,
Jake
Jiri Pirko
maintainerRe: [PATCH net-next 7/9] devlink: allow registering parameters after the instance
In reply to Jakub KicinskiQuoted 29 lines · Fri, Jan 06, 2023 at 07:34:00AM CET, [email protected] wrote:
It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.
Signed-off-by Jakub Kicinski <[email protected]>---
net/devlink/leftover.c| 22 +++++++++++-----------1 file changed, 11 insertions(+), 11 deletions(-)net/devlink/leftover.c +7
-1@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,WARN_ON (cmd !=DEVLINK_CMD_PARAM_NEW && cmd !=DEVLINK_CMD_PARAM_DEL && cmd !=DEVLINK_CMD_PORT_PARAM_NEW && cmd !=DEVLINK_CMD_PORT_PARAM_DEL ); -ASSERT_DEVLINK_REGISTERED (devlink); + +/* devlink_notify_register() / devlink_notify_unregister() +* will replay the notifications if the params are added/removed +* outside of the lifetime of the instance. +*/ +if (!devl_is_registered (devlink)) +return ; msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL );if (!msg)
net/devlink/leftover.c int devlink_params_register(struct devlink *devlink, · line 10918
Quoted 4 lines
net/devlink/leftover.c +0
-1@@ -10915,8 +10921,6 @@ int devlink_params_register(struct devlink *devlink,const struct devlink_param *param = params;int i, err; -ASSERT_DEVLINK_NOT_REGISTERED (devlink);
Hmm, params list is not protected by any lock. The protection it used was that it is static after registration. You changed it but didn't add the lock. All param register/unregister functions need to be renamed to devl_* and assert instance lock. I will fix this.
[..]
Jakub Kicinski
maintainerPATCH 8/9 netdevsim: rename a label
In reply to Jakub Kicinskierr_dl_unregister should unregister the devlink instance. Looks like renaming it was missed in one of the reshufflings.
--- drivers/net/netdevsim/dev.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/netdevsim/dev.c +2 -2
@@ -1563,7 +1563,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); err =nsim_dev_dummy_region_init (nsim_dev, devlink);
@@ -1629,7 +1629,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err_params_unregister:devlink_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params)); -err_dl_unregister: +err_resource_unregister:devl_resources_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs);
Signature · 1 line
2.38.1
Re: [PATCH net-next 8/9] netdevsim: rename a label
Quoted 5 lines · Fri, Jan 06, 2023 at 07:34:01AM CET, [email protected] wrote:
err_dl_unregistershould unregister the devlink instance. Looks like renaming it was missed in one of the reshufflings.Reviewed-by Jacob Keller <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 9/9 netdevsim: move devlink registration under the instance lock
In reply to Jakub KicinskiTo prevent races with netdev code accessing free devlink instances move the registration under the devlink instance lock. Core now waits for the instance to be registered before accessing it.
--- drivers/net/netdevsim/dev.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/net/netdevsim/dev.c +8 -3
@@ -1556,10 +1556,14 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) goto err_devlink_unlock; } - err =nsim_dev_resources_register (devlink); + err =devl_register (devlink);if (err) goto err_vfc_free; + err =nsim_dev_resources_register (devlink); +if (err) + goto err_dl_unregister; + err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err)
@@ -1607,7 +1611,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:
@@ -1631,6 +1634,8 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) ARRAY_SIZE(nsim_devlink_params)); err_resource_unregister:devl_resources_unregister (devlink); +err_dl_unregister: +devl_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs); err_devlink_unlock:
@@ -1668,7 +1673,6 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)struct nsim_dev *nsim_dev =dev_get_drvdata (&nsim_bus_dev->dev);struct devlink *devlink =priv_to_devlink (nsim_dev); -devlink_unregister (devlink);devl_lock (devlink);nsim_dev_reload_destroy (nsim_dev);
@@ -1677,6 +1681,7 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)devlink_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));devl_resources_unregister (devlink); +devl_unregister (devlink);kfree (nsim_dev->vfconfigs);kfree (nsim_dev->fa_cookie);devl_unlock (devlink);
Signature · 1 line
2.38.1
Re: [PATCH net-next 9/9] netdevsim: move devlink registration under the instance lock
Quoted 3 lines · Fri, Jan 06, 2023 at 07:34:02AM CET, [email protected] wrote:
To prevent races with netdev code accessing free devlink instances move the registration under the devlink instance lock. Core now waits for the instance to be registered before accessing it.
This sentense sounds a bit confusing to me. "to be registered" does not sound correct.
"Core now waits for the instance lock to be unlocked before executing
access the instance" perhaps would be more accurate?
Quoted 2 lines
Signed-off-by Jakub Kicinski <[email protected]>
One way or another. Code looks fine:
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 10 lines · On Thu, 5 Jan 2023 22:33:53 -0800 you wrote:
Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.
Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.
[...]
Here is the summary with links:
- [net-next,1/9] devlink: bump the instance index directly when iterating
https://git.kernel.org/netdev/net-next/c/d77278196441 - [net-next,2/9] devlink: update the code in netns move to latest helpers https://git.kernel.org/netdev/net-next/c/7a54a5195b2a - [net-next,3/9] devlink: protect devlink->dev by the instance lock https://git.kernel.org/netdev/net-next/c/870c7ad4a52b - [net-next,4/9] devlink: always check if the devlink instance is registered https://git.kernel.org/netdev/net-next/c/ed539ba614a0 - [net-next,5/9] devlink: remove the registration guarantee of references https://git.kernel.org/netdev/net-next/c/9053637e0da7 - [net-next,6/9] devlink: don't require setting features before registration https://git.kernel.org/netdev/net-next/c/6ef8f7da9275 - [net-next,7/9] devlink: allow registering parameters after the instance https://git.kernel.org/netdev/net-next/c/1d18bb1a4ddd - [net-next,8/9] netdevsim: rename a label https://git.kernel.org/netdev/net-next/c/5c5ea1d09fd8 - [net-next,9/9] netdevsim: move devlink registration under the instance lock https://git.kernel.org/netdev/net-next/c/82a3aef2e6af
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Jiri Pirko
maintainerQuoted 11 lines · Fri, Jan 06, 2023 at 07:33:53AM CET, [email protected] wrote:
Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.
Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.
This fixes the problem of the netdev code accessing devlink instances before they are registered.
Nice work. Thanks!
Quoted 29 lines · Jakub Kicinski
Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.
Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.
This fixes the problem of the netdev code accessing devlink instances before they are registered.
RFC: https://lore.kernel.org/all/[email protected]/
- rewrite the cover letter
- rewrite the commit message for patch 1
- un-export and renamedevl_is_alive
- squash the netdevsim patchesJakub Kicinski (9):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protectdevlink->devby the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
devlink: allow registering parameters after the instance
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock
Quoted 8 lines
drivers/net/netdevsim/dev.c | 15 +++-- include/net/devlink.h | 2 + net/devlink/core.c | 121 ++++++++++++++++-------------------- net/devlink/devl_internal.h | 28 ++++----- net/devlink/leftover.c | 64 ++++++++++++------- net/devlink/netlink.c | 19 ++++-- 6 files changed, 137 insertions(+), 112 deletions(-)
The whole series looks good to me. It looks like Jiri also has some planned followups that will clean some of this up even more, which is great.