Skip to contentSkip to conversation

devlink: remove the wait-for-references on unregister

PATCH Applied Jakub Kicinski 5 participants 45 messages Jan 6, 2023 – Jan 16, 2023

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

cover devlink: remove the wait-for-references on unregister

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.

This fixes the problem of the netdev code accessing devlink instances before they are registered.

RFC: https://lore.kernel.org/all/[email protected]/
- rewrite the cover letter
- rewrite the commit message for patch 1
- un-export and rename devl_is_alive
- squash the netdevsim patches

Jakub Kicinski (9):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protect devlink->dev by the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
devlink: allow registering parameters after the instance
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock

 drivers/net/netdevsim/dev.c |  15 +++--
 include/net/devlink.h       |   2 +
 net/devlink/core.c          | 121 ++++++++++++++++--------------------
 net/devlink/devl_internal.h |  28 ++++-----
 net/devlink/leftover.c      |  64 ++++++++++++-------
 net/devlink/netlink.c       |  19 ++++--
 6 files changed, 137 insertions(+), 112 deletions(-)
Signature · 1 line
2.38.1
author maintainer
#3 lore raw

PATCH 2/9 devlink: update the code in netns move to latest helpers

devlink_pernet_pre_exit() is the only obvious place which takes the instance lock without using the devl_ helpers. Update the code and move the error print after releasing the reference (having unlock and put together feels slightly idiomatic).

---
 net/devlink/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
net/devlink/core.c +4 -3
index 88c88b8053e2..d3b8336946fd 100644
@@ -299,15 +299,16 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	 */
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
-		mutex_lock(&devlink->lock);
+		devl_lock(devlink);
 		err = devlink_reload(devlink, &init_net,
 				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
 				     DEVLINK_RELOAD_LIMIT_UNSPEC,
 				     &actions_performed, NULL);
-		mutex_unlock(&devlink->lock);
+		devl_unlock(devlink);
+		devlink_put(devlink);
+
 		if (err && err != -EOPNOTSUPP)
 			pr_warn("Failed to reload devlink instance into init_net\n");
-		devlink_put(devlink);
 	}
 }
 
Signature · 1 line
2.38.1
author maintainer
#11 lore raw

PATCH 5/9 devlink: remove the registration guarantee of references

In reply to Jakub Kicinski

The objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because devlink_unregister() waits for all references to be released, meaning that devlink_unregister() can't itself be called under the instance lock.

To avoid this issue devlink_register() was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error until devlink_register() is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).

Weaken the guarantees of the devlink references.

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.

This implies that releasing of the "main" reference of the devlink instance moves from devlink_unregister() to devlink_free().

---
 include/net/devlink.h       |  2 ++
 net/devlink/core.c          | 64 ++++++++++++++++---------------------
 net/devlink/devl_internal.h |  2 --
 3 files changed, 30 insertions(+), 38 deletions(-)
include/net/devlink.h +2 -0
index 6a2e4f21779f..425ecef431b7 100644
@@ -1647,6 +1647,8 @@ static inline struct devlink *devlink_alloc(const struct devlink_ops *ops,
 	return devlink_alloc_ns(ops, priv_size, &init_net, dev);
 }
 void devlink_set_features(struct devlink *devlink, u64 features);
+int devl_register(struct devlink *devlink);
+void devl_unregister(struct devlink *devlink);
 void devlink_register(struct devlink *devlink);
 void devlink_unregister(struct devlink *devlink);
 void devlink_free(struct devlink *devlink);
net/devlink/core.c +28 -36

113 lines not on this page · raw message

net/devlink/devl_internal.h +0 -2
index 01a00df81d0e..5d2bbe295659 100644
@@ -12,7 +12,6 @@
 #include <net/net_namespace.h>
 
 #define DEVLINK_REGISTERED XA_MARK_1
-#define DEVLINK_UNREGISTERING XA_MARK_2
 
 #define DEVLINK_RELOAD_STATS_ARRAY_SIZE \
 	(__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
@@ -52,7 +51,6 @@ struct devlink {
 	struct lock_class_key lock_key;
 	u8 reload_failed:1;
 	refcount_t refcount;
-	struct completion comp;
 	struct rcu_head rcu;
 	struct notifier_block netdevice_nb;
 	char priv[] __aligned(NETDEV_ALIGN);
Signature · 1 line
2.38.1
author maintainer
#6 lore raw

PATCH 4/9 devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---
 net/devlink/core.c          | 19 +++++++++++++++----
 net/devlink/devl_internal.h |  8 ++++++++
 net/devlink/leftover.c      | 35 +++++++++++++++++++++++++++++------
 net/devlink/netlink.c       | 10 ++++++++--
 4 files changed, 60 insertions(+), 12 deletions(-)
net/devlink/core.c +15 -4
index d3b8336946fd..c53c996edf1d 100644
@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_registered() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_registered(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/devl_internal.h +8 -0
index 6342552e5f99..01a00df81d0e 100644
@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;
 
 struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
+static inline bool devl_is_registered(struct devlink *devlink)
+{
+	/* To prevent races the caller must hold the instance lock
+	 * or another lock taken during unregistration.
+	 */
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
 #define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT	BIT(1)
net/devlink/leftover.c +29 -6
index bec408da4dbe..491f821c8b77 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < state->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_registered(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < state->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_registered(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_registered(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/netlink.c +8 -2
index 69111746f5d9..b5b8ac6db2d1 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_registered(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,
 
 	devlink_dump_for_each_instance_get(msg, state, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_registered(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Signature · 1 line
2.38.1
author maintainer
#13 lore raw

PATCH 6/9 devlink: don't require setting features before registration

In reply to Jakub Kicinski

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

---
 net/devlink/core.c | 2 --
 1 file changed, 2 deletions(-)
net/devlink/core.c +0 -2
index 7cf0b3efbb2f..a31a317626d7 100644
@@ -125,8 +125,6 @@ struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
  */
 void devlink_set_features(struct devlink *devlink, u64 features)
 {
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	WARN_ON(features & DEVLINK_F_RELOAD &&
 		!devlink_reload_supported(devlink->ops));
 	devlink->features = features;
Signature · 1 line
2.38.1
author maintainer
#1 lore raw

PATCH 1/9 devlink: bump the instance index directly when iterating

In reply to Jakub Kicinski

xa_find_after() is designed to handle multi-index entries correctly. If a xarray has two entries one which spans indexes 0-3 and one at index 4 xa_find_after(0) will return the entry at index 4.

Having to juggle the two callbacks, however, is unnecessary in case of the devlink xarray, as there is 1:1 relationship with indexes.

Always use xa_find() and increment the index manually.

---
 net/devlink/core.c          | 31 +++++++++----------------------
 net/devlink/devl_internal.h | 17 ++++-------------
 2 files changed, 13 insertions(+), 35 deletions(-)
net/devlink/core.c +9 -22
index 371d6821315d..88c88b8053e2 100644
@@ -91,16 +91,13 @@ void devlink_put(struct devlink *devlink)
 		call_rcu(&devlink->rcu, __devlink_put_rcu);
 }
 
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t))
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
 {
-	struct devlink *devlink;
+	struct devlink *devlink = NULL;
 
 	rcu_read_lock();
 retry:
-	devlink = xa_find_fn(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
+	devlink = xa_find(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
 	if (!devlink)
 		goto unlock;
 
@@ -109,31 +106,21 @@ devlinks_xa_find_get(struct net *net, unsigned long *indexp,
 	 * This prevents live-lock of devlink_unregister() wait for completion.
 	 */
 	if (xa_get_mark(&devlinks, *indexp, DEVLINK_UNREGISTERING))
-		goto retry;
+		goto next;
 
-	/* For a possible retry, the xa_find_after() should be always used */
-	xa_find_fn = xa_find_after;
 	if (!devlink_try_get(devlink))
-		goto retry;
+		goto next;
 	if (!net_eq(devlink_net(devlink), net)) {
 		devlink_put(devlink);
-		goto retry;
+		goto next;
 	}
 unlock:
 	rcu_read_unlock();
 	return devlink;
-}
-
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find);
-}
 
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find_after);
+next:
+	(*indexp)++;
+	goto retry;
 }
 
 /**
net/devlink/devl_internal.h +4 -13
index adf9f6c177db..14767e809178 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
  * in loop body in order to release the reference.
  */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)
+
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
@@ -135,7 +126,7 @@ struct devlink_gen_cmd {
  */
 #define devlink_dump_for_each_instance_get(msg, state, devlink)		\
 	for (; (devlink = devlinks_xa_find_get(sock_net(msg->sk),	\
-					       &state->instance, xa_find)); \
+					       &state->instance));	\
 	     state->instance++, state->idx = 0)
 
 extern const struct genl_small_ops devlink_nl_ops[56];
Signature · 1 line
2.38.1
author maintainer
#4 lore raw

PATCH 3/9 devlink: protect devlink->dev by the instance lock

In reply to Jakub Kicinski

devlink->dev is assumed to be always valid as long as any outstanding reference to the devlink instance exists.

In prep for weakening of the references take the instance lock.

---
 net/devlink/devl_internal.h | 3 ++-
 net/devlink/leftover.c      | 7 +++----
 net/devlink/netlink.c       | 9 ++++++---
 3 files changed, 11 insertions(+), 8 deletions(-)
net/devlink/devl_internal.h +2 -1
index 14767e809178..6342552e5f99 100644
@@ -131,7 +131,8 @@ struct devlink_gen_cmd {
 
 extern const struct genl_small_ops devlink_nl_ops[56];
 
-struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs);
+struct devlink *
+devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs);
 
 void devlink_notify_unregister(struct devlink *devlink);
 void devlink_notify_register(struct devlink *devlink);
net/devlink/leftover.c +3 -4
index e6d6c7f74ae7..bec408da4dbe 100644
@@ -6314,12 +6314,10 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb,
 
 	start_offset = state->start_offset;
 
-	devlink = devlink_get_from_attrs(sock_net(cb->skb->sk), attrs);
+	devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs);
 	if (IS_ERR(devlink))
 		return PTR_ERR(devlink);
 
-	devl_lock(devlink);
-
 	if (!attrs[DEVLINK_ATTR_REGION_NAME]) {
 		NL_SET_ERR_MSG(cb->extack, "No region name provided");
 		err = -EINVAL;
@@ -7735,9 +7733,10 @@ devlink_health_reporter_get_from_cb(struct netlink_callback *cb)
 	struct nlattr **attrs = info->attrs;
 	struct devlink *devlink;
 
-	devlink = devlink_get_from_attrs(sock_net(cb->skb->sk), attrs);
+	devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs);
 	if (IS_ERR(devlink))
 		return NULL;
+	devl_unlock(devlink);
 
 	reporter = devlink_health_reporter_get_from_attrs(devlink, attrs);
 	devlink_put(devlink);
net/devlink/netlink.c +6 -3
index a552e723f4a6..69111746f5d9 100644
@@ -82,7 +82,8 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
 	[DEVLINK_ATTR_REGION_DIRECT] = { .type = NLA_FLAG },
 };
 
-struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs)
+struct devlink *
+devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 {
 	struct devlink *devlink;
 	unsigned long index;
@@ -96,9 +97,11 @@ struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs)
 	devname = nla_data(attrs[DEVLINK_ATTR_DEV_NAME]);
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
+		devl_lock(devlink);
 		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
+		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
 
@@ -113,10 +116,10 @@ static int devlink_nl_pre_doit(const struct genl_split_ops *ops,
 	struct devlink *devlink;
 	int err;
 
-	devlink = devlink_get_from_attrs(genl_info_net(info), info->attrs);
+	devlink = devlink_get_from_attrs_lock(genl_info_net(info), info->attrs);
 	if (IS_ERR(devlink))
 		return PTR_ERR(devlink);
-	devl_lock(devlink);
+
 	info->user_ptr[0] = devlink;
 	if (ops->internal_flags & DEVLINK_NL_FLAG_NEED_PORT) {
 		devlink_port = devlink_port_get_from_info(devlink, info);
Signature · 1 line
2.38.1
author maintainer
#15 lore raw

PATCH 7/9 devlink: allow registering parameters after the instance

In reply to Jakub Kicinski

It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.

---
 net/devlink/leftover.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)
net/devlink/leftover.c +11 -11
index 491f821c8b77..1e23b2da78cc 100644
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;
 
 	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
 	if (!msg)
@@ -10915,8 +10921,6 @@ int devlink_params_register(struct devlink *devlink,
 	const struct devlink_param *param = params;
 	int i, err;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	for (i = 0; i < params_count; i++, param++) {
 		err = devlink_param_register(devlink, param);
 		if (err)
@@ -10947,8 +10951,6 @@ void devlink_params_unregister(struct devlink *devlink,
 	const struct devlink_param *param = params;
 	int i;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	for (i = 0; i < params_count; i++, param++)
 		devlink_param_unregister(devlink, param);
 }
@@ -10968,8 +10970,6 @@ int devlink_param_register(struct devlink *devlink,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	WARN_ON(devlink_param_verify(param));
 	WARN_ON(devlink_param_find_by_name(&devlink->param_list, param->name));
 
@@ -10985,6 +10985,7 @@ int devlink_param_register(struct devlink *devlink,
 	param_item->param = param;
 
 	list_add_tail(&param_item->list, &devlink->param_list);
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(devlink_param_register);
@@ -10999,11 +11000,10 @@ void devlink_param_unregister(struct devlink *devlink,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	param_item =
 		devlink_param_find_by_name(&devlink->param_list, param->name);
 	WARN_ON(!param_item);
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_DEL);
 	list_del(&param_item->list);
 	kfree(param_item);
 }
@@ -11063,8 +11063,6 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	param_item = devlink_param_find_by_id(&devlink->param_list, param_id);
 	if (!param_item)
 		return -EINVAL;
@@ -11078,6 +11076,8 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
 	else
 		param_item->driverinit_value = init_val;
 	param_item->driverinit_value_valid = true;
+
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(devlink_param_driverinit_value_set);
Signature · 1 line
2.38.1
author maintainer
#38 lore raw

PATCH 8/9 netdevsim: rename a label

In reply to Jakub Kicinski

err_dl_unregister should unregister the devlink instance. Looks like renaming it was missed in one of the reshufflings.

---
 drivers/net/netdevsim/dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/netdevsim/dev.c +2 -2
index b962fc8e1397..d25f6e86d901 100644
@@ -1563,7 +1563,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	err = devlink_params_register(devlink, nsim_devlink_params,
 				      ARRAY_SIZE(nsim_devlink_params));
 	if (err)
-		goto err_dl_unregister;
+		goto err_resource_unregister;
 	nsim_devlink_set_params_init_values(nsim_dev, devlink);
 
 	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
@@ -1629,7 +1629,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 err_params_unregister:
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
-err_dl_unregister:
+err_resource_unregister:
 	devl_resources_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
Signature · 1 line
2.38.1
author maintainer
#40 lore raw

PATCH 9/9 netdevsim: move devlink registration under the instance lock

In reply to Jakub Kicinski

To prevent races with netdev code accessing free devlink instances move the registration under the devlink instance lock. Core now waits for the instance to be registered before accessing it.

---
 drivers/net/netdevsim/dev.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/net/netdevsim/dev.c +8 -3
index d25f6e86d901..738784fda117 100644
@@ -1556,10 +1556,14 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 		goto err_devlink_unlock;
 	}
 
-	err = nsim_dev_resources_register(devlink);
+	err = devl_register(devlink);
 	if (err)
 		goto err_vfc_free;
 
+	err = nsim_dev_resources_register(devlink);
+	if (err)
+		goto err_dl_unregister;
+
 	err = devlink_params_register(devlink, nsim_devlink_params,
 				      ARRAY_SIZE(nsim_devlink_params));
 	if (err)
@@ -1607,7 +1611,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_LEGACY;
 	devlink_set_features(devlink, DEVLINK_F_RELOAD);
 	devl_unlock(devlink);
-	devlink_register(devlink);
 	return 0;
 
 err_hwstats_exit:
@@ -1631,6 +1634,8 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 				  ARRAY_SIZE(nsim_devlink_params));
 err_resource_unregister:
 	devl_resources_unregister(devlink);
+err_dl_unregister:
+	devl_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
 err_devlink_unlock:
@@ -1668,7 +1673,6 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 	struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
 	struct devlink *devlink = priv_to_devlink(nsim_dev);
 
-	devlink_unregister(devlink);
 	devl_lock(devlink);
 	nsim_dev_reload_destroy(nsim_dev);
 
@@ -1677,6 +1681,7 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
 	devl_resources_unregister(devlink);
+	devl_unregister(devlink);
 	kfree(nsim_dev->vfconfigs);
 	kfree(nsim_dev->fa_cookie);
 	devl_unlock(devlink);
Signature · 1 line
2.38.1
maintainer Reviewed-by on v1 1/9
#2 lore raw

Re: [PATCH net-next 1/9] devlink: bump the instance index directly when iterating

In reply to Jakub Kicinski
Quoted 10 lines · Fri, Jan 06, 2023 at 07:33:54AM CET, [email protected] wrote:

xa_find_after() is designed to handle multi-index entries correctly. If a xarray has two entries one which spans indexes 0-3 and one at index 4 xa_find_after(0) will return the entry at index 4.

Having to juggle the two callbacks, however, is unnecessary in case of the devlink xarray, as there is 1:1 relationship with indexes.

Always use xa_find() and increment the index manually.

maintainer Reviewed-by on v1 3/9
#5 lore raw

Re: [PATCH net-next 3/9] devlink: protect devlink->dev by the instance lock

In reply to Jakub Kicinski
Quoted 6 lines · Fri, Jan 06, 2023 at 07:33:56AM CET, [email protected] wrote:

devlink->dev is assumed to be always valid as long as any outstanding reference to the devlink instance exists.

In prep for weakening of the references take the instance lock.

maintainer Reviewed-by on v1 4/9
#7 lore raw

Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski
Quoted 100 lines · Fri, Jan 06, 2023 at 07:33:57AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---

net/devlink/core.c          | 19 +++++++++++++++----
net/devlink/devl_internal.h |  8 ++++++++
net/devlink/leftover.c      | 35 +++++++++++++++++++++++++++++------
net/devlink/netlink.c       | 10 ++++++++--
4 files changed, 60 insertions(+), 12 deletions(-)
net/devlink/core.c +15 -4
index d3b8336946fd..c53c996edf1d 100644
@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_registered() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_registered(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/devl_internal.h +8 -0
index 6342552e5f99..01a00df81d0e 100644
@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;
 
 struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
+static inline bool devl_is_registered(struct devlink *devlink)
+{
+	/* To prevent races the caller must hold the instance lock
+	 * or another lock taken during unregistration.
+	 */
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
 #define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT	BIT(1)
net/devlink/leftover.c +4 -0
index bec408da4dbe..491f821c8b77 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < state->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}

net/devlink/leftover.c devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, · line 7816

Quoted 4 lines
net/devlink/leftover.c +1 -0
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_registered(devlink)) {

Good. I have patchset to remove this and linecard lock prepared. That makes things smoother.

Quoted 101 lines
net/devlink/leftover.c +24 -6
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < state->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_registered(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_registered(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/netlink.c +8 -2
index 69111746f5d9..b5b8ac6db2d1 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_registered(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,
 
 	devlink_dump_for_each_instance_get(msg, state, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_registered(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Signature · 1 line
2.38.1
maintainer Reviewed-by on v1 5/9
#12 lore raw

Re: [PATCH net-next 5/9] devlink: remove the registration guarantee of references

In reply to Jakub Kicinski
Quoted 25 lines · Fri, Jan 06, 2023 at 07:33:58AM CET, [email protected] wrote:

The objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because devlink_unregister() waits for all references to be released, meaning that devlink_unregister() can't itself be called under the instance lock.

To avoid this issue devlink_register() was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error until devlink_register() is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).

Weaken the guarantees of the devlink references.

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.

Quoted 5 lines

This implies that releasing of the "main" reference of the devlink instance moves from devlink_unregister() to devlink_free().

maintainer Reviewed-by on v1 6/9
#14 lore raw

Re: [PATCH net-next 6/9] devlink: don't require setting features before registration

In reply to Jakub Kicinski
Quoted 6 lines · Fri, Jan 06, 2023 at 07:33:59AM CET, [email protected] wrote:

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

I have a patch prepared to remove this entirely. Until then, this is fine.

maintainer Reviewed-by on v1 7/9
#16 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jakub Kicinski
Quoted 15 lines · Fri, Jan 06, 2023 at 07:34:00AM CET, [email protected] wrote:

It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.

---
net/devlink/leftover.c | 22 +++++++++++-----------

1 file changed, 11 insertions(+), 11 deletions(-)
net/devlink/leftover.c +0 -0
index 491f821c8b77..1e23b2da78cc 100644
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&

net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272

Quoted 10 lines
net/devlink/leftover.c +7 -1
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;

This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.

#42 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:

Quoted 10 lines · On Thu, 5 Jan 2023 22:33:53 -0800 you wrote:

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.

[...]

Here is the summary with links:
- [net-next,1/9] devlink: bump the instance index directly when iterating

    https://git.kernel.org/netdev/net-next/c/d77278196441
  - [net-next,2/9] devlink: update the code in netns move to latest helpers
    https://git.kernel.org/netdev/net-next/c/7a54a5195b2a
  - [net-next,3/9] devlink: protect devlink->dev by the instance lock
    https://git.kernel.org/netdev/net-next/c/870c7ad4a52b
  - [net-next,4/9] devlink: always check if the devlink instance is registered
    https://git.kernel.org/netdev/net-next/c/ed539ba614a0
  - [net-next,5/9] devlink: remove the registration guarantee of references
    https://git.kernel.org/netdev/net-next/c/9053637e0da7
  - [net-next,6/9] devlink: don't require setting features before registration
    https://git.kernel.org/netdev/net-next/c/6ef8f7da9275
  - [net-next,7/9] devlink: allow registering parameters after the instance
    https://git.kernel.org/netdev/net-next/c/1d18bb1a4ddd
  - [net-next,8/9] netdevsim: rename a label
    https://git.kernel.org/netdev/net-next/c/5c5ea1d09fd8
  - [net-next,9/9] netdevsim: move devlink registration under the instance lock
    https://git.kernel.org/netdev/net-next/c/82a3aef2e6af

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
maintainer Reviewed-by on v1 9/9
#41 lore raw

Re: [PATCH net-next 9/9] netdevsim: move devlink registration under the instance lock

In reply to Jakub Kicinski
Quoted 3 lines · Fri, Jan 06, 2023 at 07:34:02AM CET, [email protected] wrote:

To prevent races with netdev code accessing free devlink instances move the registration under the devlink instance lock. Core now waits for the instance to be registered before accessing it.

This sentense sounds a bit confusing to me. "to be registered" does not sound correct.

"Core now waits for the instance lock to be unlocked before executing
access the instance" perhaps would be more accurate?

Quoted 2 lines

One way or another. Code looks fine:

maintainer
#43 lore raw
In reply to Jakub Kicinski
Quoted 11 lines · Fri, Jan 06, 2023 at 07:33:53AM CET, [email protected] wrote:

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.

This fixes the problem of the netdev code accessing devlink instances before they are registered.

Nice work. Thanks!

#8 lore raw

Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski
Quoted 63 lines · Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---
 net/devlink/core.c          | 19 +++++++++++++++----
 net/devlink/devl_internal.h |  8 ++++++++
 net/devlink/leftover.c      | 35 +++++++++++++++++++++++++++++------
 net/devlink/netlink.c       | 10 ++++++++--
 4 files changed, 60 insertions(+), 12 deletions(-)
net/devlink/core.c +15 -4
index d3b8336946fd..c53c996edf1d 100644
@@ -67,6 +67,15 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_registered() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +309,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_registered(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/devl_internal.h +0 -0
index 6342552e5f99..01a00df81d0e 100644

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93

Quoted 8 lines
net/devlink/devl_internal.h +5 -0
@@ -86,6 +86,14 @@ extern struct genl_family devlink_nl_family;
 
 struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
+static inline bool devl_is_registered(struct devlink *devlink)
+{
+	/* To prevent races the caller must hold the instance lock
+	 * or another lock taken during unregistration.
+	 */

Why not just lockdep_assert here on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like the linecards_lock as opposed to the instance lock?

Thanks,
Jake

Quoted 131 lines
net/devlink/devl_internal.h +3 -0
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
 #define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT	BIT(1)
net/devlink/leftover.c +29 -6
index bec408da4dbe..491f821c8b77 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < state->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_registered(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < state->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_registered(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_registered(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_registered(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/netlink.c +8 -2
index 69111746f5d9..b5b8ac6db2d1 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_registered(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -211,7 +212,12 @@ int devlink_nl_instance_iter_dump(struct sk_buff *msg,
 
 	devlink_dump_for_each_instance_get(msg, state, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_registered(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Reviewed-by on the v1 series
#44 lore raw
In reply to Jakub Kicinski
Quoted 29 lines · Jakub Kicinski

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Instead the devlink instance refcount will only ensure that the memory of the instance is not freed. All places which acquire access to devlink instances via a reference must check that the instance is still registered under the instance lock.

This fixes the problem of the netdev code accessing devlink instances before they are registered.

RFC: https://lore.kernel.org/all/[email protected]/
- rewrite the cover letter
- rewrite the commit message for patch 1
- un-export and rename devl_is_alive
- squash the netdevsim patches

Jakub Kicinski (9):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protect devlink->dev by the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
devlink: allow registering parameters after the instance
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock

Quoted 8 lines
 drivers/net/netdevsim/dev.c |  15 +++--
 include/net/devlink.h       |   2 +
 net/devlink/core.c          | 121 ++++++++++++++++--------------------
 net/devlink/devl_internal.h |  28 ++++-----
 net/devlink/leftover.c      |  64 ++++++++++++-------
 net/devlink/netlink.c       |  19 ++++--
 6 files changed, 137 insertions(+), 112 deletions(-)

The whole series looks good to me. It looks like Jiri also has some planned followups that will clean some of this up even more, which is great.

author maintainer
#9 lore raw

Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered

In reply to Jacob Keller
Quoted 10 lines · On Fri, 6 Jan 2023 09:03:18 -0800 Jacob Keller wrote:

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93

Quoted 5 lines
net/devlink/devl_internal.h +5 -0
+static inline bool devl_is_registered(struct devlink *devlink)
+{
+	/* To prevent races the caller must hold the instance lock
+	 * or another lock taken during unregistration.
+	 */  

Why not just lockdep_assert here on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like the linecards_lock as opposed to the instance lock?

Yup, as discussed on the RFC - removing the regions lock specifically is quite tricky.

author maintainer
#17 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jiri Pirko
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3 -1
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
Quoted 8 lines

net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272

Quoted 4 lines
net/devlink/leftover.c +4 -0
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;  

This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.

Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S

maintainer
#10 lore raw

Re: [PATCH net-next 4/9] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski
Quoted 12 lines · Fri, Jan 06, 2023 at 10:19:34PM CET, [email protected] wrote:
Quoted 10 lines · On Fri, 6 Jan 2023 09:03:18 -0800 Jacob Keller wrote:

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 93

Quoted 5 lines
net/devlink/devl_internal.h +5 -0
+static inline bool devl_is_registered(struct devlink *devlink)
+{
+	/* To prevent races the caller must hold the instance lock
+	 * or another lock taken during unregistration.
+	 */  

Why not just lockdep_assert here on the instance lock? I guess this comment implies that another lock could be used instead but it seems weird to allow that? I guess because of things like the linecards_lock as opposed to the instance lock?

Quoted 2 lines

Yup, as discussed on the RFC - removing the regions lock specifically is quite tricky.

I will submit in a jiff. Will add the assert here as a part of the patchset.

maintainer
#18 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jakub Kicinski
Quoted 18 lines · Fri, Jan 06, 2023 at 10:22:51PM CET, [email protected] wrote:
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3 -1
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
Quoted 8 lines

net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272

Quoted 4 lines
net/devlink/leftover.c +4 -0
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;  

This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.

Quoted 3 lines

Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S

Hmm, let me explore that path. Thanks!

#19 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jakub Kicinski
Quoted 18 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:
Quoted 8 lines
net/devlink/leftover.c +3 -1
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
Quoted 8 lines

net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272

Quoted 4 lines
net/devlink/leftover.c +4 -0
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;  

This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.

Quoted 3 lines

Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S

I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.

maintainer
#20 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 21 lines · Tue, Jan 10, 2023 at 01:21:18AM CET, [email protected] wrote:
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 6 Jan 2023 13:55:53 +0100 Jiri Pirko wrote:

net/devlink/leftover.c static void devlink_param_notify(struct devlink *devlink, · line 5272

Quoted 12 lines
net/devlink/leftover.c +7 -1
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;  

This helper would be nice to use on other places as well. Like devlink_trap_group_notify(), devlink_trap_notify() and others. I will take care of that in a follow-up.

Quoted 7 lines
Quoted 3 lines

Alternatively we could reorder back to registering sub-objects after the instance and not have to worry about re-sending notifications :S

I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.

I'm confused. You want to register objects after instance register?

author maintainer
#21 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 5 lines · On Tue, 10 Jan 2023 17:35:35 +0100 Jiri Pirko wrote:
Quoted 3 lines

I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

maintainer
#22 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 8 lines · Tue, Jan 10, 2023 at 09:22:22PM CET, [email protected] wrote:
Quoted 5 lines · On Tue, 10 Jan 2023 17:35:35 +0100 Jiri Pirko wrote:
Quoted 3 lines

I did find it convenient to be able to do both pre and post-registering, but of the two I'd definitely prefer doing it post-registering, as that makes it easier to handle/allow more dynamic sub-objects.

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

maintainer
#37 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jakub Kicinski
Quoted 29 lines · Fri, Jan 06, 2023 at 07:34:00AM CET, [email protected] wrote:

It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.

---
net/devlink/leftover.c | 22 +++++++++++-----------

1 file changed, 11 insertions(+), 11 deletions(-)
net/devlink/leftover.c +7 -1
index 491f821c8b77..1e23b2da78cc 100644
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_registered(devlink))
+		return;
 
 	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
 	if (!msg)

net/devlink/leftover.c int devlink_params_register(struct devlink *devlink, · line 10918

Quoted 4 lines
net/devlink/leftover.c +0 -1
@@ -10915,8 +10921,6 @@ int devlink_params_register(struct devlink *devlink,
 	const struct devlink_param *param = params;
 	int i, err;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);

Hmm, params list is not protected by any lock. The protection it used was that it is static after registration. You changed it but didn't add the lock. All param register/unregister functions need to be renamed to devl_* and assert instance lock. I will fix this.

[..]

author maintainer
#23 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jiri Pirko
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

#24 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

#25 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

Thanks

maintainer
#26 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

Quoted 7 lines

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Perhaps during reload we should emulate complete fini/init notification flow to the user?

author maintainer
#32 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Leon Romanovsky
Quoted 7 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.

I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.

#27 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jiri Pirko
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 8 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Perhaps during reload we should emulate complete fini/init notification flow to the user?

"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.

Thanks

#33 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jakub Kicinski
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.

Quoted 6 lines

I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.

The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.

If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.

Thanks

#34 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.

Right. I think its easier to simply require devlink to be registered first.

Quoted 13 lines
Quoted 6 lines

I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.

The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.

Quoted 4 lines

If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.

Thanks

Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").

Thanks,
Jake

#35 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 11 lines · Jacob Keller
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.

Back then, we had only one special case - netdevsim. I still think that all recent complexity that was brought to the devlink could be avoided if we would change netdevsim to behave as HW driver (remove sysfs).

Quoted 1 line

Right. I think its easier to simply require devlink to be registered first.

devlink_register() is no more than a fancy way to say to the world: "I'm ready to accept commands". Right now, when the need_lock flag is removed from all devlink commands, we can place devlink_register() at any place.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines

I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.

The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.

If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.

Quoted 6 lines
Quoted 1 line

Thanks

Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").

Exactly.

Quoted 3 lines

Thanks,
Jake

maintainer
#28 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Leon Romanovsky
Quoted 48 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 47 lines · Jiri Pirko
Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

Quoted 7 lines

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Quoted 6 lines
Quoted 2 lines

Perhaps during reload we should emulate complete fini/init notification flow to the user?

"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.

Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.

Quoted 2 lines

Thanks

maintainer
#36 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Leon Romanovsky
Quoted 46 lines · Fri, Jan 13, 2023 at 07:45:46AM CET, [email protected] wrote:
Quoted 11 lines · Jacob Keller
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 12 Jan 2023 09:07:43 +0200 Leon Romanovsky wrote:

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

To be clear my preference would be to always construct the three from the root. Register the main instance, then sub-objects. I mean - you tried forcing the opposite order and it only succeeded in 90-something percent of cases. There's always special cases.

Back then, we had only one special case - netdevsim. I still think that all recent complexity that was brought to the devlink could be avoided if we would change netdevsim to behave as HW driver (remove sysfs).

Quoted 1 line

Right. I think its easier to simply require devlink to be registered first.

devlink_register() is no more than a fancy way to say to the world: "I'm ready to accept commands". Right now, when the need_lock flag is removed from all devlink commands, we can place devlink_register() at any place.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines

I don't understand your concern about user experience here. We have notifications for each sub-object. Plus I think drivers should hold the instance lock throughout the probe routine. I don't see a scenario in which registering the main instance first would lead to retry/sleep hacks in user space, do you? I'm talking about devlink and the subobjs we have specifically.

The term "dynamic object addition" means for me what driver authors will be able to add objects anytime in lifetime of the driver. I'm pretty sure that once you allow that, we will see zoo here. Over time, you will get everything from .probe() to workqueues. The latter caused me to write about retry/sleep hacks.

If you success to force everyone to add objects in .probe() only, it will be very close to what I tried to achieve.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Thanks

Yea. I was initially thinking of something like that, but I've convinced myself that its a bad idea. The only "dynamic" objects (added after the initialization phase of devlink) should be those which are triggered via user space request (i.e. "devlink port add").

Exactly.

And reload as well.

Quoted 4 lines
Quoted 4 lines

Thanks,
Jake

#29 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

In reply to Jiri Pirko
Quoted 52 lines · Jiri Pirko
Quoted 51 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 49 lines · Jiri Pirko
Quoted 34 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

Quoted 7 lines

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Perhaps during reload we should emulate complete fini/init notification flow to the user?

Quoted 7 lines
Quoted 3 lines

"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.

Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.

Listen yes, however it is not clear if notification sequence should mimic fini/init flow.

Thanks

Quoted 3 lines
Quoted 3 lines

Thanks

maintainer
#30 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 57 lines · Sun, Jan 15, 2023 at 09:35:57AM CET, [email protected] wrote:
Quoted 56 lines · Jiri Pirko
Quoted 54 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 8 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Perhaps during reload we should emulate complete fini/init notification flow to the user?

"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.

Quoted 6 lines
Quoted 3 lines

Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.

Listen yes, however it is not clear if notification sequence should mimic fini/init flow.

Well, it makes sense to me. Why do you think it should not?

Quoted 6 lines

Thanks

Quoted 3 lines
Quoted 3 lines

Thanks

#31 lore raw

Re: [PATCH net-next 7/9] devlink: allow registering parameters after the instance

Quoted 58 lines · Jiri Pirko
Quoted 57 lines · Sun, Jan 15, 2023 at 09:35:57AM CET, [email protected] wrote:
Quoted 56 lines · Jiri Pirko
Quoted 54 lines · Thu, Jan 12, 2023 at 08:58:58PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 40 lines · Thu, Jan 12, 2023 at 08:07:43AM CET, [email protected] wrote:
Quoted 26 lines · Jacob Keller
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Jan 2023 10:32:13 +0100 Jiri Pirko wrote:
Quoted 3 lines
Quoted 1 line

I'm confused. You want to register objects after instance register?

+1, I think it's an anti-pattern.

Could you elaborate a bit please?

Mixing registering sub-objects before and after the instance is a bit of an anti-pattern. Easy to introduce bugs during reload and reset / error recovery. I thought that's what you were saying as well.

I was thinking of a case where an object is dynamic and might get added based on events occurring after the devlink was registered.

But the more I think about it the less that makes sense. What events would cause a whole subobject to be registerd which we wouldn't already know about during initialization of devlink?

We do need some dynamic support because situations like "add port" will add a port and then the ports subresources after the main devlink, but I think that is already supported well and we'd add the port sub-resources at the same time as the port.

Quoted 6 lines

But thinking more on this, there isn't really another good example since we'd register things like health reporters, regions, resources, etc all during initialization. Each of these sub objects may have dynamic portions (ex: region captures, health events, etc) but the need for the object should be known about during init time if its supported by the device driver.

As a user, I don't want to see any late dynamic object addition which is not triggered by me explicitly. As it doesn't make any sense to add various delays per-vendor/kernel in configuration scripts just because not everything is ready. Users need predictability, lazy addition of objects adds chaos instead.

Quoted 8 lines
Quoted 1 line

Agree with Jakub, it is anti-pattern.

Yeah, but, we have reload. And during reload, instance is still registered yet the subobject disappear and reappear. So that would be inconsistent with the init/fini flow.

Perhaps during reload we should emulate complete fini/init notification flow to the user?

"reload" is triggered by me explicitly and I will get success/fail result at the end. There is no much meaning in subobject notifications during that operation.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Definitelly not. User would trigger reload, however another entity (systemd for example) would listen to the notifications and react if necessary.

Listen yes, however it is not clear if notification sequence should mimic fini/init flow.

Well, it makes sense to me. Why do you think it should not?

After all this years, I still don't understand the mandate of devlink reload. It doesn't load/unload driver completely and as such not really performs probe/remove sequences. There is no requirement from the driver to do anything even close to fini/init too.

Sometimes, devlink reload behaves as fini/init, but not always.

This is why I'm not sure.

Thanks

Quoted 7 lines
Quoted 7 lines

Thanks

Quoted 3 lines
Quoted 3 lines

Thanks